Runes of Magic US / AU

FuzzyDIYCE is/will be an add-on that allows the creation of skill lists that can be triggered, via a macro, in a context sensitive manner similar to DIYCE and DIYCE 2.0 does. FuzzyDIYCE is inspired by those earlier add-ons but adds the ability to edit the skill rotations and other DIYCE components in-game (as well as a few other features).

The main features of FuzzyDIYCE are:

• In-game graphical skill rotation with full drag-and-drop capabilities.
  
• Just-in-Time cached data reading so no piece of information need be read more than once.
  
• Shareable code and skill lists.
  
• In-game help system.
  
• Simplified DIYCE timer usage.
  
• Skill sub-lists.
  

Project Status:[INDENT]Phase I[INDENT]Project Startup and Design Phase.[/INDENT]

Phase II[INDENT]Core Development
Compiler and GUI Development
[/INDENT]
[/INDENT]


Table of Contents:[INDENT]Design Notes[INDENT][post=580776]The FuzzyDIYCE global environment[/post] (see also test results)
[post=580644]Data Caching[/post]
[post=580762]Data Caching part 2[/post]
[post=580653]FuzzyDIYCE Commands[/post]
[post=580702]Rejected Ideas[/post]
[post=581643]An Overview of the Design[/post]
[post=582617]Thoughts on the execution pipe[/post]
[post=582935]Conclusions From Test Results[/post]
[post=582943]Thoughts on Localization[/post]
[post=583412]Some cached data vars[/post]
[post=583706]More thoughts on localization[/post]
[post=583865]A Couple More Commands[/post]
[/INDENT]

Core Development[INDENT][post=584050]Core Pre-Alpha Released[/post]
[post=585053]Core Alpha Released[/post]
[post=585086]FuzzyDIYCE Core Loading Procedure[/post]
[post=603069]Some Design Notes for the "Compiler"[/post]

[/INDENT]
[/INDENT]

Design Notes: Data Caching

One of the big features of FuzzyDIYCE is the ability to edit skill rotations and even some custom code directly in game. This means that we'll need to have a way to compile the code into something Lua can actually execute.

For sections of Lua code, this is fairly straightforward. We can use the Lua function loadstring to compile the code and save the resulting function for when we actually need to run that code (likely do the compiling at add-on startup so as to minimise FuzzyDIYCE execution time during play).

For data variables things get a little more complex. Ultimately what we desire is the ability to read a variable that has the information we need (your target's health for example) but without having the game constantly re-ask the game servers for this information. So a type of data caching system will be needed. Though this means we need a method to intercept all accesses to the data variables. We can accomplish this caching system by using Lua metatables, and a proxy table.

In Lua, all tables can have a special metatable associated with it. This metatable is yet another Lua table that can have special entries telling Lua what to do under certain conditions. The two conditions we are interested in are when a new table entry index is created, and when a table index is accessed (see Programming in Lua, Section 13.4 - Table Access Metamethods).

There is a bit of a catch though. We always want to intercept all accesses to our data variable table and therefore we cannot allow creation of an index into this table or the index will be access without calling the metamethod. For this we use a proxy table instead. The proxy table remains empty at all times and the data is saved in yet another table.

Here's a bit of code that demonstrates what I'm getting at:


Once the above is done, all accesses to DataVars will be re-directed to the code in the metatable. The actual data will be held in the _DataVars table.

Using this technique, we can then create a caching system with custom code. Well okay, we'll need to do a bit of fiddling around to get custom code compiled and a variable to tell us if what is in the cache is valid, but that's easy stuff.

Fuzzy DIYCE Commands

Here are the commands (or at least some of the commands) I'm thinking will be needed in FuzzyDIYCE. By commands I mean the things that FuzzyDIYCE actually executes when processing a skill list. Some of the command names are chosen in order to make it easier for non-programmers.

These commands would appear much more graphical in-game, though a text version would also be needed for forum posts etc.

Skill use (try)[INDENT]
The try command is the meat of the skill list and is directly related to the skill list entries in DIYCE 2.0. The with timer part is optional.[/INDENT]


Target Manipulation (target)[INDENT]
Select or clear your current target. Target selection can be the next or previous, enemy or friend, with special selection criteria for only players, no pets, or a custom targeting function (function to call must be provided)[/INDENT]


Conditional execution (when)[INDENT]
The otherwise part is optional[/INDENT]


Function Execution (lua)[INDENT]
The lua command is provided as a means to call custom code. There is still some thoughts needed on how to handle return values.[/INDENT]


Cache Manipulation (invalidate)[INDENT]
Since FuzzyDIYCE will be caching the data variables it uses, there will be a need to force FuzzyDIYCE to re-read a data variable (for example, after selecting a new target, all data variables pertaining to the target will now be invalid). The invalidate command allows this mechanism.[/INDENT]


Sub-List Execution (perform)[INDENT]
Allows using a second skill-list as if it was part of the current list.[/INDENT]

Rejected Ideas

Here are a few ideas that I had that I've decided to reject with reasons they've been rejected. So don't ask for anything in this list.


Key bound triggers for starting FuzzyDIYCE
[indent]Though it would be nice to have, to implement this would either require Runewaker to add in some interface code so that an add-on can create its own key bindings, or changes to a special game file that would allow the keybinding functionality.

The first isn't likely to happen (though I have thought of what functionality would be needed), while the second is dangerously close to a ToS violation (if not actually violating it). I can do this last since I did something along those lines for my old KeyBinder add-on, but the possible ToS violation prohibits me from continuing that add-on as well as implementing something similar in FuzzyDIYCE.[/indent]


Time delayed actions
[indent]Nope, not going to happen. Not only would this require a boat load of fiddling around trying to get it to work, the result would be so close to a bot, or allowing activities that are very close to botting, that I simply refuse to add it in.

For legitimate uses of delayed actions, one can simply use a macro's /wait command instead.[/indent]


Lua like or full Lua support for the main/all skill list definitions
[indent]Since one of the intents for the GUI interface is to make things easier for non-programmer types to create their skill lists, implementing this would run contrary to that design decision.

Since the design of FuzzyDIYCE will have the ability to call Lua functions directly as well as implement a plug-in system for more custom functionality, there really is no need for an all Lua skill list anyway.[/indent]

Yup, the timer stuff will be pretty much ripped off from the DIYCE 2 code, but since you'd be making the skill entries in a GUI, I can see the fact that a timer is being used and so automatically create a timer for it (of course, timer management will also be implemented).

The try command is pretty much the DIYCE 2 skill entry but with a friendlier interface. I'm hoping to expand on it so that custom stuff can be added for what can be "tried". For example, I'm thinking of rolling the targeting command into this instead of an independent command, this way conditional targeting would be possible. Also would add in an assist part so that you can target what another party/raid member is targeting etc.

As to what I can use help on right now, would be the types of data needed/desired. The obvious ones are player health, target health, mana, if target is a boss/elite/mob that kind of thing. Looking at some of the current DIYCE stuff can give me a fair bit, but there is so many DIYCE variants and class combos that I can't think of everything.

Data Caching part 2

Here is pretty much the code for the caching system. It may prove to be insufficient as the design advances, but this should suffice for now.



The way this is setup, you can't just write a value directly into the DataVars table, it is assumed that it is for cached data and so the __newindex metamethod wants to see either a Lua function that will retrieve the data required, or a string contain the Lua code to do the same.

The advantage with this is that adding new (or other) data variables becomes nothing more than setting the function via the DataVars table.

For example, to create a cached data variable for retrieving the target's health percentage, you could create it with:


That's it. The rest is taken care of by the system.


Edit:
A quick note about incrementing the cache ID number. I'd originally thought about adding in some code that would wrap the ID back to 0 if we hit some rather large number, but that could potentially cause some trouble if the cache ID caught up to an old entry seldom used that just happened to have the same ID number. But a bit of research and some simple math shows that this is not needed.

Lua can handle integer numbers as large as 100,000,000,000,000 (1x10¹⁴) without errors. A number that big means that if you could press the key to start FuzzyDIYCE every tenth of a second, and did this 24 hours a day, 365 days a year non-stop, You could keep doing this for over 100,000 years and still not reach that number.

I think we can safely ignore wrapping the cache ID.

The FuzzyDIYCE Code Environment

In order to help alleviate polluting the global namespace, FuzzyDIYCE will create and use its own global environment, though access to the normal global table will of course be desirable.

Since the game client doesn't actually clear the Lua environment when you exit to the character screen then log back in (either with the same or different character), some extra processing will be needed to handle this case.

We can create our own separate global environment with a call to setfenv giving a parameter of 1 and a table we wish to use as the new global environment (the 1 parameter tells Lua we are changing the current execution context's environment. That is, the add-on itself).

But before we set the environment, we'll need to set the new environment table to be able to access the real global environment or we'll not be able to access any other Lua functions. This can be done in a couple of ways.

The first is to simply set an _G entry in our table that points to the real _G. This will certainly give us access, but requires us to always use _G.xxxx to access anything. This could be problematic when dealing with frame templates and the like.

The second method we can use is to leverage the __index metamethod to access the real _G global table if an entry isn't found in the current environment. This has the advantage that all global accesses will be completely transparent while still giving us the ability to create our "global" variables within our non-global environment. This is the method I'll be using for FuzzyDIYCE.

So here's the code for setting up FuzzyDIYCE's environment

I've been thinking of the ways to do the execution pipe for FuzzyDIYCE.

Currently, the ideas are

• Interpreted.
  [indent]This would be fairly straightforward. But the drawback is that it would be slower.[/indent]
  
• Compiled Individual Lines.
  [indent]This would be faster than just interpreted, but it does complicate the editing.[/indent]
  
• Completely Compiled.
  [indent]This would be the fastest for execution, though it complicates the editing even more. It would also require more memory since both the source and compiled versions of the code would need to be kept in memory.[/indent]
  

Any thoughts guys?

Player editing of their skill rotations.

Since everything is done in a GUI, to "execute" the actual list, the options boil down to "interpreting" the data as input by the player (slower, but easier to handle), convert each "line" to some executable code (faster, but each line must be kept in both source and compiled form), or completely converting the player's "code" to an executable function (fastest for execution, but editing and debugging become more complex, and of course both versions must be held in memory).

I'm thinking the last would be best at the end of the day for the speed of execution, but it does complicate things. Also means having a "compiler" for the player code.

Nope, the editing will be quite different. That's part of the point of all this. It's not just a text editor in the game, but a graphical/icon based editor. I am planning to allow custom functions (the lua command is one example) as well as a plugin system for more options, and other customization options.

Anyway, the editing will be a radical departure from previous DIYCE. Overall the functionality will be the same (if not better), but the editing/creation of the skill list and execution of it needs to handle two aspects. First, it needs to be easier for non-programmers to handle (this is where the graphical editing come into play) and second it should be done in-game.

The Basic Design (a broad view)

Up to this point, I've been kind of procrastinating with the actual design of FuzzyDIYCE. I've got a bit of a plan in my head and though I've mentioned some of it, there is nothing really written down for people to grasp. Sorry I've delayed this and caused some confusion. Now this isn't a formal design document, more of a discussion of some of the issues that needs to be addressed. It is all subject to change of course, but we need a starting point.

When designing something like this, we need to keep in mind two different perspectives simultaneously. First is how the user will use and interact with the add-on, and the second is how the internals of the add-on will operate.

The main aspect of FuzzyDIYCE is the graphical editor. From the user's standpoint, we need to make this as friendly and easy to use as possible. As the main point is to edit skill rotation lists, it is desirable to have the ability to simply drag a skill/item/action bar button/etc unto the editor and have it add an appropriate command for it. Obviously, the user would then need to fill in the conditions and other aspects needed, but there would not be a need to get the skill name correct since the editor can grab all those details from the item dropped in.

It is also desirable to allow the user to reorder the rotation list by dragging things around in the editor, as well as having both context help (in a seperate window) and tooltips for the editable parts.

Finally, the user should be able to share their skill rotations with others players, both in-game and on the forums. More advanced users should also have the ability to expand the add-on itself as well as create more complex skill rotations for specific tasks/situations. These modifications should also be sharable.

Looking at the above from the standpoint of how the add-on will work internally, this does present a few challenges that need to be addressed. The drag and drop part wouldn't be very difficult since the items we would be interested in for a skill rotation can all be handled via the game's cursor item and other standard functions. The reordering of the skill list would be a little more complex, but mostly because we need to handle the case ourselves, not because the activity is difficult.

Where things get a little more difficult is in the saving and loading of the skill lists, as well as how to execute the skill rotations. Since add-ons cannot save data to any old file, all skill rotations will need to be saved to either the standard SaveVariables or SaveVariablesPerCharacter data saving schemes. As this has the potential of creating some very large SaveVariables files (which may cause some problems under some conditions), a method to import skill lists from a static Lua file should also be present (static here means that the file does not, in fact cannot change during a game session). Lists saved to the SaveVariables files should override ones imported from a static Lua file so that editing a list that was imported from a static Lua file will still work correctly on subsequent game sessions.

Since the add-on cannot save skill lists in a graphical fashion, some conversion will need to occur between the graphical editor and the list that is actually saved. The current thought is to use a string/token based system for actually saving the skill list. This conversion should be fairly straightforward to implement and would also give use the data needed in order to share skill lists in a forum posts or with other players in-game via the game's mail system (a simple cut-and-paste operation would suffice for this). It would also be possible to share skill rotations by sending/receiving skill lists via the game chat system.

During loading of the skill lists, another conversion to an actual executable format will be required. Though it would be possible to use the text/token based version for performing the skill rotation, this is deemed much too slow. Therefore, the skill list will be converted into a Lua function in string format, then compiled via Lua's loadstring function. This compiled version will be what is actually executed by FuzzyDIYCE. This does complicate the add-on since all commands will require appropriate Lua code for the conversion, but a side benefit of doing it this way is that there is now a somewhat convenient way to extend the command list for FuzzyDIYCE as well as extending the functionality of existing commands.

For the execution of the actual skill list, there are a few problems that need to be addressed apart from the conversion to a Lua function. The first problem we'll need to handle is what I call the early out issue. The way current DIYCE works, once a skill has been triggered the rest of the skill rotation is skipped. However, in FuzzyDIYCE things are a little different since the skill list isn't held in an array but actual commands inside the function and therefore we require a method to skip the remainder of the skill rotation.

One way to efficiently handle this situation is to have all skill rotations as sub-lists. That is, there is one main function that calls one (or more) skill rotation lists. If any of these sub-lists triggers a skill, then the remainder of that list and any subsequent calls to perform sub-lists are skipped. This does imply that most commands can only be present in the main function while sub-lists would only allow the try and perform commands.

Thoughts on the FuzzyDIYCE Execution Pipe

Been a little while since I posted in here due to Christmas and all. Anyway, I've been thinking about the details of FuzzyDIYCE's command execution pipe. As mentioned in the previous post, some commands such as try and perform require the ability to be skipped if a previous skill has been triggered. The following is what I'm thinking is a nice way to implement this detail.

First, every FuzzyDIYCE command will be implemented as a Lua function and these functions will be held in table for easy access. This will allow easy command expansion since we can add entries into this table. This still leaves us with a need to intercept calls to the commands that need skipping, but there is an elegant way to handle this.

Effectively, we'll be using Lua metamethods to allow us to intercept function calls for the commands that need skipping and abort or ignore the actual call when appropriate. We can do this via Lua's __call metamethod. This metamethod is actually used to intercept calls using a table as if they were functions (commonly known as functables). So for commands that may require skipping over, we'll actually create a table that contains the real function and an associated metatable with a __call metamethod to do the work for us.

Here's a somewhat simple version of what I'm thinking about

Note how the skipable command is defined here. We are creating a table with an entry of func that contains the command's real function and adding the SkipMT as the metatable.

Try the above out. If you try to call Commands.normal() you will get the normal function as expected. Calling Commands.skipable() will indirectly call the function held in Commands.skipable.func via the __call metamethod. If you then set Skip to true, and try calling Commands.skipable(), nothing will happen (setting Skip back to false will allow the function to be called again).

Of course, it does make the creation of the skip-able commands a bit more complex, but we can provide a function to aid in that respect.

Quoted from "ShaWalker;582671"

As for the memory footprint with source code and compiled code, dont worry.
But if you do worry, build a string table
With it all menu's, tooltips, functions, etc derive from same table, thus sourcecode eventualy will be a table of integers.
Taking this approach will be awesome for editing and modifying.

Though I have thought of this string table thing, it doesn't work for all situations. The big hitch is when trying to send code to someone else (or having the code in a format for posting on the forums). Having just numbers would be extremely tedious and not very obvious as to what is going on. Also, there would need to be some kind of guarantee that the numbers represent the same strings on each persons computer. This may not always be the case since I want to implement a plugin system as well as allowing command extensions.

Further, what about the names of skills? Embedded custom Lua code? These things would need to remain in string format since a given skill may not be present in someone else's skill book (not being high enough level or the list is for a different class combo just for example). The embedded Lua code would need to remain a string for obvious reasons.

As to memory footprint, imagine someone with multiple characters and many skill lists for differing situations on each of those characters. We need to store the "source" version and the fully converted version at the least. There is also the issue of storing all this stuff. We don't want to make the SaveVariables table too big or bad things can happen (just look through the Technical Support forum to find posts of people having this very issue).

My current thought on handling this is to do a kind of compromise where instead of numbers, I'd use string tokens for the parts that can be converted and the rest can stay as string literals. This tokenized version would reduce the overall data size.

I also plan to have the user code be loadable from a "module" instead of just from the SaveVariables. This would make the code somewhat static, so edited versions would then need to be saved back to the SaveVariables until the user is happy with the result. They could then manually put this new version in the module, freeing up the SaveVariables again.

It is annoying that we don't have the ability to save/load to arbitrary files, but I understand why Runewaker disallows it (it is potentially dangerous after all).

Quoted from "Dellenn;582815"

I don't understand about 9/10ths of what you write Peryl,

Well this is the development thread :p

Quoted from "Dellenn;582815"

but I think the gist is, you want to make DiYCE easy for code-nubs like me, with a GUI and plugins and easy stuff we can deal with?

That is indeed the ultimate aim of all this. But also make DIYCE (well FuzzyDIYCE) easier to expand on (via plugins and extensions). And also to allow sharing of all of this stuff in-game as well as on the forums.

Quoted from "Dellenn;582815"

I <3 u Peryl

You're welcome