Showing posts with label Programming. Show all posts
Showing posts with label Programming. Show all posts

Friday, March 01, 2024

Project 21 - Code Doodle - Swarm Doodle (Introduction)

It’s been a while since I’ve built anything interesting in the Artificial Intelligence field and with all of the neural network / transformer / LLM nonsense going on right now, I wanted to go back to one of the parts of AI I love, swarm systems.

If you haven’t heard the term before, boids are a generic model for the way organisms swarm together. Think fish in a school or birds in a flock. Each animal has a pretty simple set of things its trying to do but sometimes the overall effect can be breath taking; think of a huge murmuration of starlings at twilight. Boids -- think they fly like a boid -- are a way to model how bunch of simple agents can create a very complex behaviour, and how small tweaks to an agents behaviour can change the way a whole system performs (think starlings compared to geese). They're a very old addition to the world of artificial intelligence, and they've also been extreamly important to the world of film making.

Walter Baxter / A murmuration of starlings at Gretna / 

That being said, it’s been a very long time since since I’ve messed around with swarms (or boids) and I’m quite rusty and -- as I think is a theme here -- my vector math always seems to be just a touch insufficient. So, I’m going to write a short series as I go about putting my swarm system together.

I’m not going to go into all of the details right now, but I’m going to roughly follow Craig Reynolds Boid model. A lot of the details aren’t in the original paper and the content in the Wikipedia article is slightly out of line with my memory, so I’m going to bushwack from what I have to a hopefully working swarm system (rather than do more reading right now).

For this doodle, I’m taking my pre-existing moving agent code -- which I always wrote with an eye to doing a swarm system and expanding on it. To start I have little circles that try to balance being between the mouse and the middle of the screen. To build boids we’re going to have to expand that a little and add in a few more things to do.

I’m trying to stay close to Reynolds model for now, although there’s lots of fun to play with later on. I’m going to set up three “urges” for the boids, a separation urge -- “I’d like to not crash”, a heading (or alignment) urge -- “I want to go where everyone else is going” and a cohesion urge -- “I don’t want to be the only one out where I can get eaten”.

I’ve been working on this for a while -- and then not working on it for a while, so I’m starting to write with the intention of kicking myself into finishing the project. I’m about halfway done putting everything together -- which you'll notice when you look at the code in the repository, and I think I’ll write this up in four or five posts over the next few weeks. Once I get the basics done, I’m looking forward to all the other things I can play with.

Saturday, April 08, 2023

A General Update on Code Doodling

Doodle Code is my “knitting” project, where I dabble in programming while watching TV or during family gatherings (hey, it works for my family), and I haven’t had any really goals, so I haven’t had a point to stop and update. It also turns out I’ve been a bit forgetful and messy, but oh well. Now seems like time to dust it off and share it with anyone who’s interested, even if it's not the repository of useful teaching code I'd thought it might be originally.

Graph of Languages for the Github repository, showing Java at 77.1%, Processing at 18.8% and Python at 4.1%.
Can you tell I'm a programmer who did most of his learning in the late 90s and early 2000s?

There are about 15 doodles, in various states of completion. Dunking on the dumb mathematical things sports casters say has been a bit of a theme and beyond that playing with colours. Everything else is things I thought would be interesting or just stuff I’ve been meaning to do (like the Coding Train Challenges).

I’d drifted away from doodling for a while, so back in February I thought it was time to start again. Then I came to the realisation that basically nothing was organised, documented or finished. I’ve started poking at that, but as it turns out it’s more fun to do new things than it is to go and clean up my own mess.

Graph of GitHub Commits showing tjkendon with 251 commits from January 2022, to April 2023, 6739 lines of code added and 1464 lines of code removed.
Github is always good for that endorphin rush of numbers go up.

Each doodle now has at least a description and a Readme, some are better documented than others, but that’s a starting point. I’m trying to strike a balance between working on new stuff and cleaning up what’s there so that it might be useful for someone else at some point. It may not be good for my existing habit of not getting stuff done, but I’m honestly feeling quite happy doodling around with stuff and I’m not feeling that compelled to “finish” anything. I am slowly learning to unlink my feelings of self-worth and happiness from productivity and I must say it’s quite refreshing and the project is leaving me feeling pretty good even if it may not be that actually useful.

Monday, February 28, 2022

Feelings about Set Similarity

The first tiny program I worked on for Code Doodle was a list difference calculator. It takes two lists and tells you which elements are on both lists or how similar the lists they are. 

I made this mostly to back up my joke that every time you watch any World Cup coverage an English sportscaster will talk about 1966 and how well the English team has done against whoever they're playing since then, despite the fact that ... none of the players are the same. Similarly the record of two NHL teams must be discussed even when there are more players who are playing for the other team than played for the same team after some period of time.  I thought having a program which backs up how ridiculous these types of statements are might be entertaining. As it happens, I also ran into a problem at work that needed me to find which people were on two lists, so it's not just a joke, but it's still mostly a joke.

All this program does is calculate the union, intersection and set difference for the two lists and then calculate how similar the two lists are. This is a bit messy in Java, due to the limitations of the Java Collections API (certainly the Python version of this program would be tiny), but nothing particularly hard to do. It was a fun project, so I wrote it up, wrote up some basic tests, and it seemed good enough.

Later on, when I was adding in the ability to read from a file, I put in two lists that were different than my original lists and thought I must have made a mistake. Even though I thought the two lists should be 50% similar, the program told me that they were 33% similar. 

So I went and checked through my code, my tests and my math. All of those were correct, as for as I could see, so I did what naturally comes next, I asked Google. After a few journeys through Stack Overflow and Wikipedia I managed to get something of an answer. I'm not weird, math is weird. 


Well, I am weird and math is weird.


Different Types of Set Similarity


My intuition when I was writing the program said to me that the similarity of two sets would be the size of the intersection of those sets, divided by the size of the union. If there are 10 total things and 3 of them are on both lists then the two lists should are 30% similar.

The problem came for me looking at two sets like this:

{a, b, c, d} and {c, d, e, f}

The intersection {c, d} has 2 elements and the union has 6 elements {a, b, c, d, e, f}, so by my reckoning above, the two sets are 33% similar. When I came back to it, my intuition said since each list has half unique elements and half common they should be 50% similar. So I had two "intuitive" results, both of which seemed to be mathematically correct, but which "felt" right in different contexts.

I should probably have done some reading first, but this wasn't a reading first kind of project. Once I did start reading I was at least pleased to discover that I'd done my math right.


Jaccard Coefficient


For my first calculation of similarity, I had stumbled upon the Jaccard coefficient (or the Jaccard index), which is the ratio of the size of the intersection to the size of the union.


A diagram of letters a - f with a - d grouped in a pink circle and d - f grouped in a blue circle.

Mathematical Notation: A = {a, b, c, d}, |A| = 4, B = {c, d, e, f}, |B| = 4

Mathematical Notation: Intersection of A and B = {c ,d}, Size of Intersection of A and B = 2, Union of A and B = {a, b, c, d, e ,f}, Size of Union of A and B = 6


Mathematical Notation: Jaccard coeefficient is the size of the Intersection of A and B divided by the size of the Union of A and B. This is equal to 2 divided by 6, which simplifies to 1 divided by three or 0.33



The Jaccard Coefficient has you figure out  the percentage of elements that are in both sets out of the all of the possible elements. This makes sense for a lot of sense for a lot of applications, for example if you want to see how similar two photos are for example you can see how many of the pixels are the same and the more similar pixels the more similar the photos should be. Because it focuses on on the intersection compared to all elements it makes the most sense when you're considering the whole space of the elements, rather than a specific set of those elements.

The problem here is that it doesn't suit my primary goal, which is to be able say that 22% of the 2014 English Men's football team returned to play in the 2018 English Men's football team. Instead the Jaccard coefficient tells me me that 5 out of the 44 people who played football for England in 2014 and 2018 played in both.So while it's an accurate measure of similarity for a lot of purposes, it doesn't tell us as much about the overlap between the two sets we're looking at.


Sørensen–Dice coefficient 

I learned all that about the Jaccard Similarity while I started searching for the "right answer". Did I do the math wrong? No. Did I build a meaningless measure? Not that either. Thankfully after that I found the Wikipedia entry that introduced the Sørensen–Dice coefficient. For Sørensen–Dice, instead of looking at the elements in the intersection compared to all the elements, we're looking at the size of the intersection compared to the average size of the two sets.


Mathematical Notation: Sørensen–Dice coefficient is = 2 times the size of the intersection of A and B divided by sum of the sizes of A and B. This is equal to 2 times 2 divided by 4 plus 4, which resolves to 4 divided by 8, which simplifies to 1 divided by two or 0.5



Since we're now saying that 2 out of the 4 elements in either of the sets are the same, we get much closer to that feeling I was looking for. Now if we have 5 out of the 22 members on the 2014 English Men's football team repeat as 5 out of the 22 members on the 2018 English Men's football team, then the two teams are 22 % similar, as opposed to the 12% Jaccard would suggest. 

For our example we now have 2 of the 4 in each set so a 50% similarity as opposed to the 33% similarity from the Jaccard coefficient and I think this produces a more intuitive measure for the kind of application I'm intending here. 


Szymkiwicz-Simpson Coefficient

While I was at it, I also discovered the Szymkiwicz-Simpson coefficient, which is also called the Overlap coefficient. This is almost the same as the Sørensen–Dice coefficient, but instead of - effectively - looking at the average size, we're comparing the size of the intersection to the size of the smaller set. This reframes the question a little bit into how much of the smaller set appears in the larger set.


Mathematical Notation: The Symkiwicz-Simpson coefficient is equal to the size of the intersection of A and B divided by the minimum of the size of A and the size of B, this resolves to 2 divided by the minimum of 4 and 4, which resolves to 2 divided by 4 which simplifies to 1 divided by 2 or 0.5


The big difference between Szymkiwicz-Simpson and Sørensen–Dice is that for sets with a large size difference the Sørensen–Dice will be higher. So the focus (as the name Overlap implies) is really focused on the overlap vs the overall similarity. Since for most of the data I'm playing with, the two sets are pretty similar in size there's really no effective difference between the two, but I'm sure in different applications, the difference might be really helpful. (For example: Is it more remarkable that two people were both in classes of 100, or more remarkable that they were in a class of 100 and a class of 10?)


If we take the same basic set of six elements and slightly reorganize the two sets we're comparing we can see real differences in the outcomes.


A diagram showing the letters a - f with one set containing {a, b, c} and the other set containing {b, c, d, e,f}

Mathematical Notation: Set A contains elements a, b and c, the size of set A is 3, Set B contains elements b, c, d, e, f, The size of Set B is 5.

Mathematical Notation: The intersection of Sets A and B is b and c, The size of the intersection of sets A and B is 2, The Union of sets A and B contain a, b, c, d, e, f, the size of the union is 6.

Mathematical Notation: The SS coefficient = size of the intersection of A and B divided by the minimum of the size of A and the size of B, this resolves to 2 divided by the minimum of 3 and 5, which resolves to 2 divided by three or 0.67, the SD coefficient is equal to 2 times the size of the intersection of A and B divided by the sum of the size of A and and the size of B, this resolves to 2 divided by 8, which simplifies to 0.5, the J equals the size of the intersection divided by the size of the union of A and B, which resolves to 2 divided by 6, which simplifies to 1 divided by 3 or 0.33



Feelings about Set Similarity

In short this is a silly project. It is entirely based out of me having listened to too many sports-casters say dumb statistical things. (Don't get me started on the "Data Science" of restricting events to such a ridiculous extent that you can find the most shots made by a left-handed player in the odd minutes of the second half of a game.)

I'm sure in *most* applications of similarity measures, there's something which will indicate what the right measure is for the application. Jaccard and Sørensen where both working in ecology and populations (and I think Szymkiwicz was as well) and so depending on what question they were looking at, the both make sense for comparing different populations. In computer vision, Jaccard makes a lot of sense for checking if two images are the same, but Szymkiwicz might make more sense to see if a smaller picture is part of a bigger one.

For my purposes, the primary answer is feel. Does it feel like the 2014 English Mens football team is more similar to the 2018 or the 2010? So even while math might leave me to lean on Jaccard, the other two feel like a better fit.

I actually ran a twitter poll on the topic (and thank you to all 5 people who answered ... it was a weird poll).



The results of the poll showed a split and I can't say anything at all other than to suggest that depending on your background and your purpose the way these measure "feel" is different. So when you're writing a program or analyzing data, it's worth thinking about how someone looking at your output is going to feel when the look at your results. Are they going to align with how they think about the problem and how they think about the data?

For my list difference calculator, I decided to avoid making a decision and added the options to calculate any of these three different similarity measures.






Tuesday, January 18, 2022

Project 21 - Code Doodle - Introduction

 As I've mentioned in my last few updates I'm starting to get a little bit more done on some of my *other* projects. Despite my best efforts though, some of those projects are big, and hard to finish and they need me to learn things, even when I think I've kept them small. 


The solution to that is *clearly* to start doing something else as well. Because spreading myself thinly has really been a key to my success. 


Oh.


Right.


No, it's definitely the other thing where I'm not great at finishing things and it's easier to play with the new shiny thing than it is to get shit done.


That being said, I am starting a new project and I have a bunch of ideas whey I think it's a good idea.


Ok Fine. I'll bite, what are you doing?

I'm writing a bunch of tiny programs. They may help me do other jobs or they may just be things that seem cool to play with. Basically, I want them to be simple, easily completed, code doodles.

A cartoonish drawing of the word Code surrounded my some doodles of squares, lines and toast.



You Mentioned You Had Ideas as to Why This Wasn't a Bad Idea

Aha. Actually I said I have ideas as to why this is a *good* idea. Take that, Scalziesque interlocutor!

Still.

The biggest reason I want to write tiny programs is that I want to write tiny programs. I don't write that many programs and programming is fun.Sometimes your brain is a little bit on automatic and you want to engage it just enough to to have done something. It's knitting basically. Programming, fun times.

The second reason is that I'm working on *another* project, related to teaching people to code and I think one of the things we do very poorly when programming is teaching new programmers to read other peoples' programs. I've never had time when I'm teaching to get beyond the bare basics of what I need written for the students to work with, and so this seems like a great time to just build up a repository of interesting programs so that people can *look* at them at some point. In fact, at some point, I hope to have other people contribute interesting tiny programs too. (I don't know what that point is, so if you're interested in publicly sharing the code for a tiny program, drop me a line.)

The third reason is that there's a ton of things I still need to learn. Also there's a not insignificant list of things I've forgotten to a greater or lesser extent. And much as I hope Game Tracker will be great for teaching me, it's *already* too big and complicated for what I'd hoped to do for a lot of things and I need some places to play before I build those things.

The fourth reason is just to give myself practice finishing things. I'm bad at it. I've been bad at it for most of my life and so keeping the programs tiny seems like a good way to finish them.

The fifth (and for now final) reason is that it's a fun way to run into more interesting things I haven't had a chance to think about before. For example, how similar *are* the sets of {1, 2, 3, 4} and {3, 4, 5, 6}. So the fifth reason really ties the other together a bunch of the others in a way that should hopefully keep my brain moving.

So What Are You Doing?

I've made a GitHub repository and I'm uploading stuff there. No guarantees any of it's good or interesting (and if I've solved any of your intro to programming assignments by accident, (Sorry!) I'm not doing it on purpose. - Hit me up and we'll chat assessments in programming.) Feel free to keep an eye on it if that's the kind of thing that interests you.

So far I have 2 programs in the repo.
  • List Difference - takes a look at two lists and lets you see how similar they are. It's intended to help statistically mock sports casters. This one's pretty finished.
  • Food-o-rac-o-cycle - named for the food machine on the Jetsons (I think). This will tell me what to make for breakfast. This one's off to a good start.
I'm also planning to add a couple of other programs that I'm working on to help with the Chrono Sprites. I started them separately, but they fit the model for Code Doodling, so I'm going to drop them in too.

I'll update from time to time, but if you're interested in what I'm up to looking at the repository is probably the best place to see the latest.





Tuesday, January 11, 2022

Blog: What I learned, forgot and learned again about Java Preferences

I'm not an expert in Java, but I have been using and teaching it for a really long time. One thing I've always struggled with is the right way to manage settings and preferences, especially stacking different levels of settings (defaults, setup time and run time). The various systems I worked on in grad school were massively over complicated (to the extent where I think in my masters’ system, configuration was probably a third of the whole system).

For the Game Tracker I set out to keep things really simple, and ideally to do things "the right way". Given that I don't actually work in software development with anyone other than myself, figuring out "the right way" can be a bit of a struggle sometimes. Still I'm trying to start modelling my own technology learning after Julia Even's approach and learn and focus on what I have learned.

For handling settings, I've used Java's Properties quite a bit (they made up a big portion of my PhD system, which was *much* more streamlined than my masters). Having an API to handle getting simple values in and out of a file is really useful. Preferences are an extra layer on top of that, where you hand over keeping track of the storage to the JVM.

While I'm still a little grouchy about Preferences not being transparent to the user or the programmer. They do feel like "the right way" to keep track of settings. I was especially confused when I cam back to the Game Tracker after a long break and so had to relearn a lot about Preferences. Hopefully this will be a good overview and collection of the good resources I found.


Overview

As I mentioned, preferences let you hand off worrying about where your information gets stored. All you have to do is tell Java that you would like this class’s preferences and then either read or store a value. When you end your program, Java will store everything and the next time you run your program your preferences will be there.

You can recompile or even completely delete and replace your program and your preferences will still be there. If you’ve ever deleted and then reinstalled a program and been surprised that it remembered everything from before, that’s preferences in action. (And that bit where stuff just gets magically remembered is why I’m sometimes uncomfortable with the idea of preferences, sometimes you just want a program to go away, but we’ll talk about that later.)

When we’re talking about preferences, we’re talking about the small pieces of information you need to make your program run. For example, with the game tracker the list of games is too big to go in preferences, but keeping the file name where I'm keeping the list of games makes sense. There’s not a strict limit, but keeping too much data in your preferences isn’t a great idea. Additionally, everything is handled as a string by Java so you want to avoid storing anything complex, like an object, and should keep in mind that you’ll have to handle moving numeric data back and forth.


Using Java Preferences

You access Java preferences through a java.util.prefs.Preferences object. We’ll talk about how you *get* a Preferences object shortly, but once you have an object you can treat it as a dictionary. You have a put(String key, String value) method where you can store a value under a key and a get(String key) method which gives you the last value you stored under key.

If you have other types of data, you have to handle them separately. The good news is that you don’t have to do any parsing, there are putBoolean(), putDouble(), putInt() methods and getBoolean(), getDouble(), getInt() methods (and actually there’s a pair of methods for every primitive Java type).

You get a Preferences object by asking the Preferences system for it. Similar to the way you ask System for System.out, there are static methods on the Preferences class for getting your Preferences object. Unlike System.out where there’s only one object to get back, there’s actually many possible objects.

Each package in your program gets its own preferences, so when you ask Preferences for your object, you need to give it your class so it knows which package’s preferences to give back. The static method Preferences.systemNodeForPackage(Class theClass), takes a class object and so you can statically give it a class like CLI.class. *

That there are actually two kinds of preferences. System Preferences (which is what you get if you use the method above) and User Preferences. System preferences are shared by every user on a computer, but the user preferences are unique for each user. Generally user preferences are the right place to store things unless you have a strong reason why everyone needs to share. To get the user preferences, you’ll use the static method Preferences.userNodeForPackage().

From there you’re pretty good to go. The first time someone runs your program you can store their preferences and there after you can access and update them as necessary. I’m not sure it’s the “right way” but I think you should also offer your users a chance to see what’s stored in preferences and a way to remove them (the Preference class has methods to do both). I still need to implement this in the Game Tracker, but I think transparency is good to keep in mind.


My GameTracker Example

The last time I worked on the Game Tracker I reorganized a lot of the components. At this point ** I’m really only keeping two preferences for Game Tracker; the name of the persistence manager and the  file name of the file which has the game and play session data.

The persistence manager is managed in the CLI class. As the CLI class is starting it figures out which PersistenceManager it needs and then calls a factory to return an instance of that manager.  For every preference, I keep a constant string in the class to keep the key value, and I’ll often also keep a default value as well.

When the CLI starts, it processes the command line arguments (a story for another day) and checks if the option for the persistence manager has been set. If the value has been set, the CLI saves the value in the preferences. Then the CLI consults the preferences for the name of the persistence manager (with the default). This means that the value that’s loaded is either the new value that was just passed in, the preference that was set before or the default value if it’s never been set.


    private static final String PERSISTENCE_MANAGER_PREF = "gametracker.persistence_manager";
    private static final String PERSISTENCE_MANAGER_DEFAULT = "CSV";

if (cl.hasOption("datamanager")) {
  Preferences.userNodeForPackage(CLI.class).
    put(PERSISTENCE_MANAGER_PREF, cl.getOptionValue("datamanager"));
}

persistenceManager = PersistenceManagerFactory.getManager(
  Preferences.userNodeForPackage(CLI.class).get(
    PERSISTENCE_MANAGER_PREF, PERSISTENCE_MANAGER_DEFAULT), args);

Within the CSVPersistenceManager, the only preference kept is the data file name. Like the name of the persistence manager in the CLI, I keep the name of the preference and the default value as constants, although here the preference name actually sits in the super class FilePersistenceManager.


In FilePersistenceManager:
protected static final String DATA_FILE_PERF = "gametracker.datafile";

In CSVPersistenceManager:
private static final String DATA_FILE_DEFAULT = "gametracker.csv";

Persistence managers are passed the arguments off the command line, so as in the CLI, the super class checks the arguments if there’s a value for the datafile and that is taken and loaded into the preferences.

if (cl.hasOption("datafile")){
  Preferences.userNodeForPackage(
    FilePersistenceManager.class).put(
      DATA_FILE_PERF, cl.getOptionValue("datafile"));
}

The CSVPersistenceManager then checks the preferences to find the filename for the data file.

datafile = new File(
  Preferences.userNodeForPackage(FilePersistenceManager.class).get(
    DATA_FILE_PERF, DATA_FILE_DEFAULT));

The data file can be changed from the setFile method in FilePersistenceManager.

public void setFile(String fileName) {
  Preferences.userNodeForPackage(
    FilePersistenceManagerMenu.class).put(DATA_FILE_PERF, fileName);
      datafile = new File(fileName);
}

I’m not getting a ton of use out of preferences right now. I was doing more before I started streamlining, but right at the moment those are the only two things I need to keep track of. Again, I’m not sure I’m “doing it right,” but this setup seems to work for me. 


The Magic Inside

As I said, the great thing about preferences is that the JVM and the operating system really take care of it for you. So if you’re happy to trust the “magic system” to keep stuff up and running, then you’re good to go. On the other hand if you’re trying to figure out what the hell is going on, then let’s talk about it.

I ended up researching this because when I came back to the GameTracker after a long break, I had no idea what was going on or why the software (which I wrote) knew where the files were without me telling it (also sometimes I’m bad at taking notes - but I did have an issue on the topic for myself). This was especially a problem because I was looking for a local configuration file, which just wasn’t there.

I set out to figure out what was going on. Thankfully a number of people have written pretty good explanations of the preferences system. I decided that I could add a little bit by bringing those articles together and the best way to learn anything is always to teach it (or in this case blog about it).

So with that being said, there are two things to talk about if we want to open up the magic inside.


How are the preferences organized?

At the JVM level, all preferences are kept in trees. There’s the system tree and a tree for each user. (So as a programmer you have access to two trees, the system and the current user). 

Each node in the tree represents a package. The root of the tree is the empty package and it’s named “/”.  Then every package below the root is its own node. If we take my GameTracker as an example, the CLI.java main class is in package ca.kendon.gametracker.cli. That means that the root node has a child node, “ca/”, which has a child node “kendon/”, which has a child node “gametracker/”, which has a child node “cli/”.


A tree consisting of a single branch with nodes "/", "ca/", "kendon/", "gametracker/", "cli/"
A tree diagram of the CLI class's preference.



Since we have multiple packages with preferences (.cli and .core)  both of those packages's preferences would be held under their common parent.  so "gametracker/" would be the parent to both both "cli/" and "core/"


A tree diagram consisting a branch with nodes "/", "ca/", "kendon/", "gametracker/", which has two child nodes "cli/" and "core/"
A tree diagram showing the common ancestors for both the "cli/" and "core/" packages.



Notice that the names of the preference node have those slashes at the end. If you ask for the full name of the node the CLI preferences are in it will be "/ca/kendon/gametracker/cli”. If that seems like a file name... well that might be important later (in some contexts).


Where are the preferences stored?

The JVM has its nice abstraction of preferences in trees, but that doesn’t help when your question is “where is that value *actually* stored?” Now we actually have to take a look under the JVM and see what the operating system is up to. Preferences exist in all operating systems, not just for Java but for any software that needs to remember things. Of course, since all operating systems have their own ideas about where stuff is, there’s a different answer for every operating system. For a quick overview we’ll look at Windows, MacOS and Linux.

Victor Kropp has a really good post on the details.

Windows 10

As windows is wont to do with all of its configuration information, things are stored in the registry. 

User preferences are found at HKEY_CURRENT_USER\Software\JavaSoft\Prefs

System preferences are found HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Prefs

Remember that you can look at the registry using the Registry Editor (regedit) utility, which is installed by default in Windows.

MacOS

On MacOS, preferences are stored in the Library. Since we have two types of preferences, system and user, MacOS actually has two types of libraries (system and user) for preferences.

System preferences are found in /Library/Preferences/ in a file called com.apple.java.util.prefs.plist

Each user’s preferences are found in their home directory in ~/Library/Preferences/ in a file *also* called com.apple.java.util.prefs.plist

plists are Apple’s favourite way to store things in a way that slightly frustrates everyone … well it frustrates me certainly. They’re XML files (often compressed) which can be a pain to actually look at. On the command line you can run the command:


defaults read ~/Library/Preferences/com.apple.java.util.prefs.plist


Which will show you the preferences for your own user.  For example mine includes:


        "gametracker/" =         {
            "cli/" =             {
                "gametracker.persistence_manager" = "CSV";
            };
            "core/" =             {
                "data_file" = "data/games.data";
}; };

If this seems a little overbuilt it's the weird collision of Apple's plists and xml files which always end up having some weird structures.

Linux


On Linux, preferences for each node are stored in its own file (in case you’d wondered why the Java name for preferences looked so much like a file name).

System Preferences are stored starting in the directory /etc/.java/.systemPrefs/

User preferences are stored in each user’s home directory under ~/.java/.systemPrefs/

Under those directories, the JVM builds a directory structure that matches the preference nodes, with the leaf nodes being files. In my Game Tracker Example, that means that the "gametracker.persistence_manager" = "CSV" preference is stored in the file ~/.java/.systemPrefs/ca/kendon/gametracker/cli

This makes a lot of sense to my *nix-centric brain and I think explains a lot about the structure they chose when setting up preferences in *nix environments.

Conclusion

I'm always wary of systems that aren't transparent to the user, but preferences provide a good way to keep track of the details about a program without making a mess or requiring a lot of overhead. I still have a lot to learn, already, I'm wondering about some of the things I'm doing to update the preferences from the program. Hopefully this overview provides a place to get started and answers a few of the questions the preferences seem to draw. 

Footnotes

* I *think* you can use the this identifier to get the class dynamically, but I'm not sure whether or not that's a bad idea. (Thus far I've found I've made mistakes using the literal class - copy and paste errors - so using this may be smarter.)


** This is a snapshot of the GameTracker from early November 2021.


Monday, November 22, 2021

Project 18: Game Tracker Update - 2021

 So it's been a long while since I've sat down and worked on the Game Tracker and it's also been a long while since I've written anything other than vague intentions for working on the project. Fortunately my new focus on creative work has push me back to it.


I was actually pushed by a post I'm writing for the blog that I'll share pretty soon. I've wanted to do a good explanation of Java's Preferences. I managed to thoroughly confuse myself with them and so thought I'd follow the Julia Evan's approach of writing about what confused me. I'd left my project in a thorough mess back in March - the equivalent of the truck sitting in the old bard with the engine out and wires and tubes laid out all over the place - and so in order to show how I was using preferences, I thought it might be a good idea to put the bits back together.


When I abandoned the project, I was in the midst of trying to convert the save system (persistence) to use JSON. That was part of a grander idea to connect to some kind of web interface - Google Sheets and Firebase are the two I've been eying. I ended up struggling to get Java and JSON to play nicely together and after a little bit of cobbling, I discovered that Joda-Time -  which I've used for managing time in Java for *years* - was also making everything complicated. 


I gave up and decided that since I mostly wanted the system up and running so I could finish the blog post - which I've been working on for months. I set up a project - well... took over the one from March - called 2021 refresh and tried to get a minimal set of the Game Tracker up and running. That was more or less fun, I built a CSV version of the data manager and then did all the cleanup I had to do to get up and running.


A screen shot of the Game Tracker showing the main menu.
Game Tracker - Pretty much the same after all these years.



I also realized that I've been keeping the GitHub repository for the project private. I think that's partly based on not feeling like the project was ready for people to see and partly fear about showing off my own programming work. Given what I've been thinking about with creative work and how I want to use my time going forward, hiding isn't going to help me, so I opened the repository to the public and put and MIT Licence on it - not that I think it will be terribly useful for anyone. I found Choose a Licence, which was helpful for comparing licences.

 

I also got to write a readme, which at the end of the day turned out to help me feel better about the whole project and make sure that anyone who stumbles across it will know what on earth is happening. Nothing about the whole project feels at all glossy or cool - like I would like it to - but it's happening and I've moved it a little bit forward. I found Make a README which was a really good resource.


I like that I can return to the idea of small tasks for the project now. For example, for reasons that are beyond me - maybe as a demo for students - I set the game systems to be a Java Enum. This is a terrible idea since for some reason new video game systems have been released in the last few years and I've even purchased some of them. It'll be nice to be able to jump in and work on that for a little bit and then be done.


I also have a lot of other things I'd like to do. I'd like to expand the interfaces to include a desktop GUI, something for my phone and something for the web. I also still want to set up remote data management for the system. I'll get to those at some point, but again my new approaching to getting things done means they'll hopefully trickle out over time.

Sunday, November 10, 2019

Project 20 - That Code Click - Introduction

I like it when things click. When I’m programming I love that moment where things go together perfectly, and I love that moment in my own brain when a topic suddenly snaps into focus. I also like when I’m working with students and I see that moment for them.


This next project is my attempt to share that feeling and to help people learn about all of the stuff in computing that I think is really cool. It’s a chance to look at how things work, how things fit together, how things were designed and how those elegant moments in problem solving come to be.

I hope this is a chance for me to stretch my writing, teaching and communication skills. Additionally there are a lot of other technical skills I should pick up such as video production and things like that. It’ll also be a nice chance to chase down those topics I’m interested in, but never really have the time to manage while I’m teaching (and then forget about when I’m not teaching).

For the short term I’m going to simply create a few written articles and upload them here while I start building out the idea. In the longer term I’d like to see a blog and then maybe a fully dedicated website to host those articles and other supporting material.

For this first iteration of the project, I’m planning to write one article on “Counting in Binary on Your Fingers” a fun trick I always enjoy using to introduce binary numbers. I’d like to get that finished some time before November 29, including text and my own photos to illustrate. I'll also put together a list of future topics I'm thinking of. If you happen to have a great computing "click" you'd like to suggest, let me know.

Sunday, November 03, 2019

Project 19 - Pong - Phase 1

Project 19 - Pong

I am implementing Pong. Yes, that Pong. I’m implementing it both for fun, but also because it’s going to be necessary for a project for work in the fairly new future.


I figured since I was implementing Pong, I might as well make a project out of it. As a bonus since my implementation is basically finished, I get a free finished project out of the deal.

I decided, as is my wont, to work in Processing.org. This is pretty much my go to platform whenever I need to prototype something or do a thing with quick interactions. I’m also educating myself about P5.js which is proving to be fun as well.

My original take was to work with vectors, which has exposed me to all of the linear algebra I didn’t learn or have forgotten. This will also be helpful for my Bubble Puzzler work (which apparently I haven't updated here, to my surprise). I then remember that my Phase 2 for this project is to have a working version of Pong on the Atari ST, and so I’d be better off handling things like coordinates and motion as simple variables. You can find the source in on GitHub.


// player positions on the screen
int p1Y;
int p2Y;
int p1X;
int p2X;

// paddle display information
int paddleW = 10;
int paddleH = 75;

// player scores
int p1Score = 0;
int p2Score = 0;

// ball position on the screen and motion
int ballX;
int ballY;
int ballMoveX;
int ballMoveY;

int ballSize = 10;

Within the program my code is fairly basic, I’m relying on processing’s control of the framerate, and basically assign the ball a speed between 2 and -2 on the y axis and 2 and -2 on the x axis. This feels like a fairly workable implementation of speed, although increasing it as the game goes on would be an option (I suspect an actually competitive game would go on for quite a while).

The ball bounces off the top wall and off the paddles. If the ball hits a paddle near the edge (about 1/8th of its total length) then it bounces in the y direction as well as the x.


void bouncePaddle(int paddleX, int paddleY) {
    // bounces the ball off the paddle 
   if (((ballX + ballSize >= paddleX) && (ballX <= paddleX + paddleW)) && 
              ((ballY + ballSize >= paddleY) && (ballY <= paddleY + paddleH))) 
        {
     ballMoveX *= -1;
     
     // reflects the ball back on the y axis if it hits near the edge of the paddle
     // mostly for fun, not sure it was in pong, but I enjoy it in most clones
     if ((ballY + ballSize < (paddleY + (paddleH / 8))) || 
         (ballY + ballSize > (paddleY + 7 * (paddleH / 8)))) {
       ballMoveY *= -1;
     }
     
   }
   
}


This produces enough interesting effects that I’m calling this phase of the project done.

That being said, in the short term, I suppose I need some more inputs because solitaire pong seems not-too much fun. Beyond that I think that’s probably it for the Processing.org implementation. My next priority is to prepare to produce the Atari ST version of the game. I would also like to produce a version of Breakout because that seems fun and possibly also add in a few interesting visual effects.

I should be done the AtariST version by early January because I need my students to start on their projects by then.

Thursday, April 25, 2019

Project 18 - Video Game Play Tracker - Update 2 (v0.2.0)



I actually finished up v0.2.0 of my game tracker months ago, but I finally got some free time so I thought I’d write up what I have. I have probably forgotten a lot of the details at this point, but I’ll just soldier on. In this update I finished up adding in aggregation and filtering of play sessions to the display system (and did a little bug fixing). 

Filtering


I wanted to be able to filter play sessions into specific time periods. Right now I start a new spreadsheet every year to keep track of the data in a manageable way, and I start a new set of tabs within each spreadsheet to track data for each month. I wanted to be able to replicate this in an ad hoc way, which would allow me to build a more informative and flexible way to look at the data.
A google sheet showing games played on the first and second of january 2019.
A small view of my tracking spreadsheet.

Additionally it’s nice to be able to include or exclude games or groups of games from the list. For example, how much time have I spent playing all the games in the Zelda franchise? I have no idea right now but filtering would make that a question I can ask easily.
As such, I added two types of filters for play sessions, one for games and one for dates. The filters work by finding all the play sessions that match and then showing only data from those sessions. 

The game filter works by matching the (whole) name of a game. Multiple games can be added to the filter which then allows multiple games. So far, I don’t have any structure like a series or a genre, so I don’t have filters working for that, but that’s something to work on in the future. The filter provides a list of all play sessions that match the set of game names.

A screen shot of a list of filters, one for Legend of Zelda: Breath of the Wild, one for games played in the last half of January 2018.
The two basic filters.


The date filtering works by using date windows. Each window has a beginning and an end date and the filter allows through all play sessions that are between the two ends. If either of the two dates is unset then the filter allows all sessions to the beginning or end of time.

It took a little fiddling to make them work, in particular to make an empty filter allow everything (which is backwards to how the filters are designed, but extremely unintuitive to work with).

Aggregation


In my spreadsheet I keep a log of all games played for the year (including the date, the game name and the amount of time played). Then for each month I break out one pivot table that includes the total amount of time each game has been played for the month and a second table that breaks out the games played for each day so that I can produce the stacked bar-chart of the games. I needed the aggregation system to allow me to basically do that in the stand alone system.

A screen shot of a google sheet showing the games played in January 2019
January's games played chart. (See a better view in my January 2019 post).


Aggregation was interesting because I wanted to make it as abstract as possible, but still needed to make it concrete enough to work in the command-line interface. I think I may need to take a second swing at it to make it really work smoothly. 

The way it’s currently constructed aggregators take a set of play sessions (raw or filtered) and extracts a aggregate, which is a map organized by game name, the type of aggregate and the value (as a double). Aggregates can be merged to put together different stats about a set of play sessions. 

A screen shot of the program showing the total time, number of times played, the mean play session and the average play session.
The aggregated games in my test file (mostly from early 2018).


As I said the system is a little messy and I think having now let some time pass I could do a better build solution.

Project Update

Compared to the first update, I did a better job of using the git workflow and breaking down my work into smaller chunks and updating regularly. This was helpful in the work I needed to do to finish my thesis project. I also tried to start teaching the workflow to my students this semester and I think generally the “do small things well” mantra has done me (and them?) well.

A listing of the main menu of the game tracker.
The current state of the Game Tracker


It’s also been months and months since I finished the actual main work for this update. I did break some tests which ended up taking me a while to fix. Since then I’ve mostly been working on my PhD so I’ve had to backburner the whole thing.

The two big directions I’d like to take the project in next, are to produce a GUI version using JavaFX and to integrate a Google Doc back end storage. The first step I think is going to be the hook up Google sheets and then go on to the JavaFX front end.

My plan, as it stands, is to finish the next update including the Google Sheets connection (and some various fixes / reorganization) by June 1, 2019.

Friday, May 18, 2018

Project 18 - Video Game Play Tracker - Update 1 (v0.1.0)

I’ve really had a lot of fun with this project so far.


For version 0.1.0 I’ve added in the basics, including:
  • A game representation, including the name, the year and the platform.
  • A set of games and get back games that match particular criteria
  • A play session representation, including the game, the date and how long the session lasted.
  • A list of play sessions
  • Managers to load and save game and play  data from and from csv files.

I’ve also built a simple text-based UI, which allows you to save, load and add games and play sessions.

The Basic Text Based UI Menu


I also think I did an ok job using the git flow workflow, although I think I ended up adding too many features in my release 0.1.0 branch. I also added a ton of tests, even if I didn’t quite get myself to a test first mentality.

So I’m happy enough with v0.1.0. RIght now I’m keeping it in a private GitHub Repository, I should probably just open it up, but right at the moment I’m not sure I want to. At least just yet.

The basic tracker running in the terminal.
In terms of usability it’s not great, but it’s a good starting point. My next goal from here is to introduce filtering and aggregation so I can see play sessions aggregated over a list and filter for specific games or time periods. I’ll post an update around July 1, 2018.

The Books I Read - November 2024

November was a bit weird. The Hands of the Emperor is long, but excedingly good. I'm continuing to find Anna Lee Huber a very engagin...