Apex F Public Resources

Post Reply
User avatar
IskatuMesk
Xel'naga World Shaper
Xel'naga World Shaper
Posts: 8328
Joined: Sat Feb 07, 2009 1:40 pm
Location: M͈̙̞͍͞ͅE̹H̨͇̰͈͕͇̫Ì̩̳CO̼̩̤͖͘ జ్ఞ‌ా

Apex F Public Resources

Post by IskatuMesk »

This thread will be devoted to research and information for the Apex F campaign project. This information is being released publicly once it is fairly solidly sorted out, in the hopes it may benefit other campaign creators on this site.

This thread assumes familiarity with the Apex project.

Introduction

Apex F is a Build and Destroy campaign. Much of the research we have conducted circulates around building an environment that supports this gameplay properly and facilitates large amounts of additional content being added at a later time.
Computer AI
Throughout my time with Starcraft 2, from the beginnings of WoL Beta to more recent days, I have pursued computer AI solutions for various projects. In the first Apex F developer video I explained various methods that were attempted by both Blizzard and ourselves. To recap -

Blizzard - Attack wave triggers spawn units and a-move them to you. Rarely used bullies for base building.
Apex A - Attack wave triggers that assigned melee wavetypes and used DefenseRadius headers. Always used bullies for base building.
Apex F Phase 1 - Modified melee AI using MapID and Player ID comparisons to determine builds. Sometimes used bullies, mostly Galaxy (SetStock).
Apex F Phase 2 - Further refined and modified Galaxy, complete with rewritten Attack Wave handling.
Apex F Type U (2015 Q4) - Refactored SetStock to be controlled using trigger calls, heavily modified core galaxy to support the hybrid setup, cleaned up Attack code

Throughout time I had JademusSreg attempting to help me. However, he was unable to achieve results with the AI, despite supposedly having great confidence in his ability to correct its many issues. His continued disappearances and lack of specifics leads me to believe he simply lost interest, as he was often attempting to overengineer his solutions. It is possible he simply may be busy elsewhere, but given lack of communication, I can only assume we are on our own.

The solutions provided come as a result of a ~2 week period in which HKS just yolo'd his way through the AI. This is likely to be the only campaign-ready AI yet created in Starcraft 2 that does what it does, given the confusion and lack of understanding I universally received when I attempted to find information regarding such matters. They may not be the best or most efficient manners of doing things, but they "work", and that's what matters currently.

Although simply copying the melee AI "works" like in Phase 1, I wanted a better solution, one that could tackle some major trouble points that yet remained.

Congo behavior

Traditionally, Melee AI uses two methods to assign units to c_waveMain and then merge that into attacks.

NewUnit* - This, located in the Racial files that then springs off to the global command, can sort units via type before assigning them to c_waveMain.
SetAttackStatus* - Triggers a merge from c_waveMain to Attack when it has a small number of max units or enough units defined in its command. An infexible command overall.

Originally, Apex A's triggered attack waves avoided the congo issues and other problems associated with melee because, clearly, they weren't melee. However, the triggers had other issues associated with assigning wave types to them. The attempts to assign wave types to triggered attack waves do not work because the waves require the hardcoded properties of the melee code in that it runs incessantly. If it doesn't fire incessantly the code eventually "runs out" of things to do, and doesn't know to "restart". The waves had a habit of standing around and doing nothing after a point. I was able to change the idle functions to attack, and this caused them to attack instead of idle, but the problem remained elsewhere; the waves never returned to defend, and only defended at all if they were being attacked when the wave first fired.

Phase 2 solution

HKS' proposed solution was to simply have c_waveMain handle itself in terms of merging. SetAttackStatus is no longer used. Instead, c_waveMain looks for a number of units in its current possession and then checks a series of attack waves to merge them into.

Code: Select all

void AIWaveMain (int player, wave w) {
    int count = AIWaveUnitCount(w);
    int state = AIWaveState(w);
    int attackState = AIState(player, e_attackState);

    // Allows c_waveMain to be used for defensive purposes.
    if (AIDefenseThreat(c_dtAnyThreat, player, w)) {
        AIDefendSelfWithWave(player, w);
        return;
    }
    else if (AIAnyAllyNeedsDefending(player, w)) {
        AIDefendAllyWithWave(player, w);
        return;
    }

    // Don't merge units if c_waveMain is in combat/defending?
    if (AIWaveIsInCombat(w)) {
        return;
    }

    // Check c_waveMain unit count if we should continue on to merge with available attack waves.
    // Stock waves are largely used because adding more wave definitions didn't work past the first.
    // MergeCount defaults to 20
    if (count >= MergeCount[player] && !Drop[player]) {
        if (!Wave0[player] || AIWaveUnitCount(Attack0) == 0) {
            DebugAIPlayerWave(player, "main: merge -> attack wave 0");
            AIWaveMerge(player, c_waveMain, c_waveAttack);
            AISetAttackState(player, e_attackState_Attack);
            Wave0[player] = true;
            return;
        }
        else if (!Wave1[player] || AIWaveUnitCount(Attack1) == 0) {
            DebugAIPlayerWave(player, "main: merge -> attack wave 1");
            AIWaveMerge(player, c_waveMain, c_waveAttack1);
            AISetAttackState(player, e_attackState_Attack);
            Wave1[player] = true;
            return;
        }

...
By bypassing the automerging behavior of SetAttackStatus, but retaining the behavior of NewUnit*, we can still sort unwanted units out of attack waves (like transports), keep the defense behavior, and merge the waves on our terms instead of whenever.

This eliminates 90% of the congo behavior and adds some more customizability to the players.

Drop Attacks

When I first asked JademusSreg about drop attacks, he described them to me as being set up in a way in which they were guaranteed to never fire. Indeed, the melee AI already has drop attack code, it just doesn't do anything - even when manually called from triggers.

Furthermore, once functional, the drop attacks had this odd problem of being unable to find targets on a map filled with undefended enemy CC's and such. I believe he has since fixed this by handing them an alternate target type. We attempted to feed the AI a trigger that found a random enemy building, but this made the game stall horribly. These attack wave triggers are extremely susceptible to performance troubles.

HKS eventually worked drop attacks into the main attack string.

Code: Select all

    else if (count >= MergeCount[player] && Drop[player]) {
        if (!Wave0[player] || AIWaveUnitCount(Attack0) == 0) {
            DebugAIPlayerWave(player, "main: merge -> drop wave 0");
            AIWaveMerge(player, c_waveMain, c_waveAttack);
            AISetAttackState(player, e_attackState_DropAttack);
            Wave0[player] = true;
            return;
        }

...
The main caveat of this system is that the player is either a drop player or he isn't. This is set via a flag in their build order script.

Drop Attacks are kind of fucky. They don't attempt to escort the transports with air units, they take a long time to load their transports, and units don't "seek out" the transports like they did in BW. So, this is not the best solution possible perhaps (assuming there is another solution at all), but when it does work, it's pretty hilarious.

Performance

Previously I was getting a solid 14ish fps on 0x01, shown in the first dev video. Pending HKS' changes, I get an average of 20-30. The zerg map still has troubles, but we're not sure why.

Many things lead to performance problems. As mentioned, the AI running through its code over and over in itself can be expensive, particularly if its running through demanding checks, like certain evaluations in the attack code.

The Clear Obstructions wave in the attack code is particularly bad. It would stall (HKS describes this as not a "stall" but some execution time) for up to 400-500ms on HKS' computer, which is significantly faster than mine. Furthermore, this code had a habit of failing to find destructible cocks to destroy, even when they were blocking their base.

Removing this code entirely, and then just assigning the cocks to a neutral player the AI treated as hostile, had a noticeable performance boost and had the intended effect working much better.

HKS gutted all difficulty-related checks (we use ChIn as our base), all low difficulty-exclusive code, and a bunch of other crap. I removed the home wave behavior and other checks from zerg.galaxy so the zerg wasn't running through so much for every larva the repopped (queens are now part of their attack waves), and other little things all contribute to very slightly higher FPS. We also removed all counter-related stuff and anything that issued Stock commands outside of the main thread (certain commands caused Terran to always build Marauders or Ravens, for example).

That said, the performance is still a big concern. Jademus once mentioned to me there is some setting for AI tick rate, but he didn't tell me what it was or where it was located. This supposedly can dramatically impact performance, but the cost would have to be tested carefully.

HKS attempted to throttle the AI's SetStock stack. While the performance gains were notable, it dramatically hurt the AI's ability to construct things in a timely manner.

Oddities

If there are two "If" statements side by side in the AI's build order, weird shit happens. Weird shit like things getting ignored. Specifically, the Zerg were having issues making Greater Spires. Ultimately I fixed it entirely by setting their expansion bonus hatcheries to use C_standard instead of C_defense for their build placement type. wat

If Danger Maps are disabled, Air units stop working for the melee AI entirely. They seem to have some hardcoded switch that prevents them from getting hit by NewUnit* or something.

An Understanding of SetStock
SetStock is the system commonly used in sc2 for AI unit and building construction. In Brood War, you hand the game a command with a priority, and then hardcoded RNG and bugs decide if it will actually attempt it. Sc2 seems to have a similar priority system, likely passed over from its warcraft 3 engine roots, but the SetStock system entirely bypasses it and seems to render the traditional build commands useless.

SetStock functions by placing a series of commands and then "enabling" them. For buildings in particular, the game will build an invisible "blueprint", not unlike an invisible bully-enabled base in my older methods, and then try to build that. It will only try a few times - SetStock must be run ad infinitum to get results (the melee AI seems to run it every tick). As one might expect, this can be costly on performance.

Particularly big bases have issues with the blueprint. There seems to be a max distance or max amount of stuff the blueprint can hold at once. I use tech segregation or time segregation to "break up" what it builds at once. This dramatically tightens base construction and alleviates many issues where the AI just wasn't making certain buildings. Example:

Code: Select all

	if (AITechCount(player, c_TB_Drydocks, c_techCountCompleteOnly) >= 3) {
	AISetStock( player, 17, c_TB_SupplyDepot );
	AISetStock( player, 14, c_TB_Starport );
	AISetStock( player, 8, c_TB_Factory );
    }
	
		if (AITechCount(player, c_TB_Drydocks, c_techCountCompleteOnly) >= 6) {
        AISetStock( player, 10, c_TB_Phalanx );
		AISetStock( player, 10, c_TB_Hoplite );
    }
I don't know if there is any real benefit to using SetStockNext, but SetStockEx lets you assign per-case BuildAI suffixes, which can determine the algorithm it uses to place its structures. However, like BuildAI itself, I think these only account for structures that are already built, and not those in the "blueprint". This makes segregating defenses especially important.

Type U (2015 Q4)

ApexU, a short explorative venture near the end of 2015, saw enormous progress in the mission to extract performance from the game's troubled prototype AI. After 3.0 was released, HKS spent a short time looking back at the then year-old AI we left behind at the death of the Apex concept in 2014. After polishing the Attack Wave code he decided to look at the SetStock stack which was believed to be responsible for much of the AI's baffling overhead and, additionally, the 1-2 second stutters that occurred throughout gametime regardless of unit counts or mapsizes.

Image
During ApexU's research time I dropped the size of 0x05, further reduced player and unit counts, and divided my performance tests into unit tests. No performance gain was had through any of it. Clearly, unit counts and pathing costs were not the issue here.

The net result of our research is that the constant "cycling" behavior of the Galaxy is responsible for a very large amount of performance loss. Tests with Exotic Gardens on b.ent yielded that difficulties, alliances, and other settings had no impact. Green Tea performs even worse than vanilla. All of them yielded hard stutters on game startup and a steady drop to single digit and, eventually, 0fps within minutes.

How and why the Melee AI relies on this perpetual on-tick cycling of SetStock, along with around a dozen other otherwise worthless Galaxy files, is about as baffling as why Metzen has a job. Nonetheless, by disabling this behavior and relegating the responsibility of tech switches/updates to triggers that call the galaxy blocks manually, significant performance gains were finally observed. The stuttering is extremely rare now and surely related to Attack Wave evals, and the FPS floor outside of combat rose as much as 30fps in many tests.

While still considered a prototype, the ApexU has significant potential in establishing a functional campaign foundation of this nature for the first time since the launch of WoL beta.


I plan to edit and add to this thread as time goes on, and HKS is also likely to contribute.
Last edited by IskatuMesk on Tue Nov 03, 2015 4:43 am, edited 3 times in total.
User avatar
IskatuMesk
Xel'naga World Shaper
Xel'naga World Shaper
Posts: 8328
Joined: Sat Feb 07, 2009 1:40 pm
Location: M͈̙̞͍͞ͅE̹H̨͇̰͈͕͇̫Ì̩̳CO̼̩̤͖͘ జ్ఞ‌ా

Re: Apex F Public Resources

Post by IskatuMesk »

Art Tools


Some quick notes about the Art Tools.

Commit is Bugged

Don't use it.

"Make DDS Textures" produces unusable m3

Use this to generate your textures if needed, then export again with "Use Mopaq Paths". The issue is that Make DDS Textures generates an m3 that looks for local paths relative to where those dds files came from on your computer. The m3 will not work for anyone else as a direct result. Textures go into assets\textures\.

- BUG - Use mopaq paths applies to preview, too. So you can't have it on when working, or preview won't attempt to load any textures.

Geometry ignored by preview gets exported

I sometimes use an untextured block to use as a 0,0,0 parent to rescale my units. This object gets ignored in preview because it lacks an sc2 material. However, exporting does NOT ignore it, unless it's hidden. This will break the m3 instantly.

Extra

- Game supports at least 64 frame sheets. Note that there's flipbook settings in both the particle and the material. You can technically use this to pull multiple animation sets from one sheet.

- Seems to support more than 116+ animations a unit so far.

- The particle editor doesn't support non-uniform scaling of particles. You have to cheat using Tail length. Tail length only works if the particle actually has velocity. A fair annoyance for my many non-uniform flares. (wtf is going through their heads?)

- To get particles to orient their face using a different origin than the center you need to use the Pinned type. My goal was to get strings of lightning where they don't appear on both sides of a spawnpoint. Boy, this took way too goddamn long to get working. In sins you can just set the axis to top right, right, bottom right, center, etc., sc2 has no such options and pinned is not likely to work with most of my other textures that need axis changes.

- Ribbons don't seem to support flipbooks, random directions, or the majority of other things that would have made them very useful for lightning-like effects.

- Cutscene editor needs to be manually set to preview at Fastest speed. Especially important for particles.

- I think the game culls entire frames when it lags. This can easily cause particles that rely on small frame windows to appear to vanish.

- Weird - The animation naming systems aren't simply a guideline, they are required. Any divergence has extremely odd behavior like animations referring incorrect entries instead or T-Posing/not animating at all.

- Hardpoints are clientside only. No sweeping breath attacks I'm afraid.

- No collision meshes except for client-side physics. I had these break a lot, too. I think physics implementation is either really buggy or half-finished overall.

- T-Pose at frame 0 is kind of a big deal.

- I use MindTex for my normal map generation (for lineage 2 and WoW models and such). So far it has produced better results than nDo did for me in the past.
User avatar
IskatuMesk
Xel'naga World Shaper
Xel'naga World Shaper
Posts: 8328
Joined: Sat Feb 07, 2009 1:40 pm
Location: M͈̙̞͍͞ͅE̹H̨͇̰͈͕͇̫Ì̩̳CO̼̩̤͖͘ జ్ఞ‌ా

Re: Apex F Public Resources

Post by IskatuMesk »

These are the units and/or textures I converted for the Protoss player color glow in Apex F. It is not a complete conversion of the game's assets. I omitted some units I planned to replace (such as the Zealot) and some of the assets are not finished (the Pylon and Nexus need tweaking). I also have NOT tested the GnaCarrier rework ingame, so the engine particles are probably too intense/big.

The Nullifier was lifted from Mapster's site (can't recall credits, feel free to look if interested) and others were either reversed using sc2printin's python script (thanks for the help with that, btw) or originated from the art tools.

Given the excessive scale of Apex F, it goes without saying these are prototypes. They may not be entirely consistent or work well in an alternate environment. But given there was a few people interested in it in the past, and no one ever rose to the challenge, I figured I'd fill in my own 5 year request.

Simply credit me/Gna respectively if you use my/his assets.

http://www.gameproc.com/meskstuff/apex_dev/Protoss.7z
http://www.gameproc.com/meskstuff/apex_dev/Textures.7z

As the project is long dead I do not provide any post-release support. These files are presented as they were in my _TEST directory and I cannot guarantee they will work out of the box or that I packed everything. If it is evident I forgot a texture or seriously fucked up something (for example I am not sure if I even have a GnaCarrier with the proper texture directories set up) that would be easy to fix, then you can post a request. I will not, however, be converting any new units or otherwise finishing these.

As a bonus, the Zeratul model has fixed bone weighting and vertexes for the arm blade. Those are seriously screwed up in vanilla.

Image

http://www.gameproc.com/meskstuff/apex_dev/Weapon.7z
http://www.gameproc.com/meskstuff/apex_dev/_Mesh.7z - Source Max Files (no textures)


Amongst others, these are prototype particle effects I made when I first started using the Art Tools back in 2014. While I had planned for so much more to be made, I never got around to trying it.

The particles may make reference to some Delphinium textures. I am not 100% sure. I am also not 100% sure what directory they should go into (probably the usual). If something isn't working be sure to check the m3 directory data in model info.

Many of the textures are hand made as seen in my FumeFX texture pack, others are pulled from various resources too numerous to recall.

Credit if used.

The missile is referring a default Sc2 texture. I remember being told they won't work unless you re-import them into your map. You'll have to check that one yourself.

The Arclight Arrow is referring an extremely large texture that was being used as a conceptual placeholder in lieu of a future, more appropriate replacement. Alter as needed.

Let me know if I forgot something. Beyond that I cannot offer any support.
User avatar
IskatuMesk
Xel'naga World Shaper
Xel'naga World Shaper
Posts: 8328
Joined: Sat Feb 07, 2009 1:40 pm
Location: M͈̙̞͍͞ͅE̹H̨͇̰͈͕͇̫Ì̩̳CO̼̩̤͖͘ జ్ఞ‌ా

Re: Apex F Public Resources

Post by IskatuMesk »

I created a video for the art tools a while ago I never got around to fully completing. Since it is highly unlikely I will be touching the game again, I have decided to release it as-is.

http://www.gameproc.com/meskstuff/apex_ ... -muxed.mkv
User avatar
IskatuMesk
Xel'naga World Shaper
Xel'naga World Shaper
Posts: 8328
Joined: Sat Feb 07, 2009 1:40 pm
Location: M͈̙̞͍͞ͅE̹H̨͇̰͈͕͇̫Ì̩̳CO̼̩̤͖͘ జ్ఞ‌ా

Re: Apex F Public Resources

Post by IskatuMesk »

Given the recent surge of sc2-related discussion on the site I figured I would post an update about some of the new developments in the Apex F research that happened recently, which gave rise to the U variant of the project. It has been over a year since I last touched Starcraft 2, but HKS, with his newfound mastery of programming from Unreal 4, managed to tackle the AI arm of the project with a renewed stretching exercise.

In a nutshell, my original hypothesis of the persistent on-tick or on-heartbeat behavior of the AI being responsible for much of its performance troubles was indeed incorrect. Specifically, the SetStock chunk of the galaxy files is responsible for some of the most severe of the performance issues, including a lot of the overhead and the ultra severe stuttering one experiences in traditional melee with the AI.

It isn't enough to throttle the SetStock stacks, you have to disable this behavior entirely and use timed calls to have sequenced build orders. This can be done via triggers. The FPS floor for some of my tests rose by 30+ FPS due to this relatively simple change. It really makes one wonder why Blizzard's intern made the AI work the way it does in the first place. After all, it gains nearly no benefit from constantly running the SetStock stack. The only real advantage you gain is on-tick access to if statements for conditional switches, but there's far more efficient and performance-conscious ways to achieve that. It's quite possibly due to the limits of galaxy itself, or the way their natives work.

In any event, we've reduced the performance hit of the AI to mostly combat-related stuff. Waves and Tactical AI must be persistent still for obvious reasons, but almost no drawbacks have been thus far observed from disabling persistent stock resets. It actually fixed a bunch of problems I was having with them.

Image
0x05, which previously had a maximum fps of 14 and a low of 4, now has a maximum of 45 and a low of 8. 0x01, which had an average of 18 to 20fps, can sustain 60 in conditions that result in single digit with vanilla AI, but performs superior in virtually every respect - attack wave handling, construction, and tactical.

Suffice to say, this was a break in AI development I've waited almost 5 and a half years for. Although most of what I wanted to do with Starcraft 2 is still plainly impossible, like Armageddon Onslaught and large FFA's, some portions of my designs are quite plausible now. It remains to be seen what exactly I may consider attempting with it, but my original desire of making the game's first true B&D campaign remains. Such was the goal of the AI research to begin with.

Also, my first attempt to get a custom portrait was a success, so there's that.

Image

I will update the OP to cover the U concept of the AI.
Post Reply