~Thirdstyle's New Judging System~

[CENTER]THIRDSTYLE’S NEW JUDGING/BATCH SYSTEM
[/CENTER]

Hello everyone! As of late, we have received many comments, both negative and positive, regarding our current ThirdStyle Judging System. For those of you that do not know what this is, let me explain:
  • A member charts a song.
  • The person sends it to a Simfile Batch Email.
  • The Moderator will post a thread and in the OP (Opening Post) they give a link to the file where it can be downloaded.
  • Members from the community come in and judge (review) the file and write notes on likes/dislikes/fixes that can be made.
  • A file *only* needs 2 acceptances from reviews to become an acceptable file, and therefore proceeded to be put in game for the community to play.

The system started out very good, however after a while some aspects started to become questionable. For instance, one flaw that many have commented on is the fact that we only need 2 acceptances to get a file in game. Many feel that this should be a higher number because it could jeopardize the quality of files being released in game. Another flaw pointed out was the way acceptances were being stated. Some members would post in a forum and say “Accepted” without any information at all. For example:

“Looked over the file, and accept all difficulties”

This is not a proper judging technique. A proper technique is given here at this link:

Halogen-‘s Review of Retribution(Jonny)

There have been more comments on other issues regarding this system, and with that being said, many of us have come together to further evaluate the system and see how we can make it much better. After doing a tremendous amount of work, and never ceasing to stop until a new system was made, we have finally come to agreement with a good, stable, sufficient system that will make judging more interactive, and make it more professional. We have taken many members' suggestions from the past and the present and have put it all down on paper and worked out a great system!

So without further-adieu we present ThirdStyle’s New Judging/Batch System!!


ThirdStyle will *Hire*, yes HIRE, 5 permanent, dedicated, and loyal members to become judges. These 5 members will have the opportunity to show that they can review files diligently and sufficiently on a scheduled time for songs to constantly be released during the course of the year.

As well, ThirdStyle will also hire 5 back-up Judges. These judges will be brought into play when:
  • A judge is unable to fulfill their duties for a certain review/batch.
  • A file that was stepped by a current Official Judge.
  • If a fellow judge passes a file for personal reasons.
Now that we have that out of the way, let’s go over the system itself.

The Judging system will be based in BATCHES. Yes, we said batches. This will work as follows:
In the Simfile Review Forum, there will be a few threads that will be created and stickied. These threads are:
  • ThirdStyle’s Batch Release/Queue Thread
  • ThirdStyle’s Batch Review Thread

  • ThirdStyle’s Batch Discussion Thread

These threads, although self-explanatory, will have certain “rules” going on in them:

Thirdstyle's Batch Release/Queue Thread - This thread will be accessible to only ADMINS or Moderators. In this thread, there will be 2 posts only. The first post (OP) will be the Judge portion where spoiler tags will be made to show the list of files sent to the email for a specific batch time, and to show that the files have been released to the Judges for review. The 2nd post will be the Queue portion where there will be spoiler tags for each batch that shows accepted files from the Judges, and files that need to be released in game.

ThirdStyle’s Batch Review Thread – This thread will be accessible to ALL judges. Here the judges will have their own Post, wherein they will write their reviews for each batch week inside of there.

ThirdStyle’s Batch Discussion Thread – This thread is accessible to EVERYONE. Anyone who has submitted a file may come in here and discuss any questions or concerns about their files reviews. Keep in mind though, that there MUST be an order of respect being maintained at ALL TIMES.

Now, the Batches will be weekly, bi-weekly, or monthly depending on the flow of files through the website. For now, we will start off with bi-weekly. So if we started up a batch review on Monday Februrary 18th, it would include all files that have yet to be judged starting on February 4th and ending on February 17th. All of the files received in that period of time will be brought up in the Batch Queue and judged accordingly. Each judge has 1 WEEK to review all the files and get their notes in. At the present time we do not have an insane amount of traffic for files, so as of now a 1 week period of time should be sufficient enough to complete notes for files. So, Batch Queue’s will be released on every other Monday. For example:

If we start on February 18th:

Batch 1 - February 18th – Including files between February 4th and February 17th.

Batch 2 - March 4th – Including files between February 18th and March 3rd.

So on and so forth. If you want a file to be admitted to a certain batch, make sure you get the file sent in to ThirdStyle before Midnight of the day of the batch review. As long as it’s submitted correctly it WILL be judged. Batch notes will be posted inside of the Batch Review Thread under each judge’s appropriately titled post.

Quick side note, while the 5 main Judges have their respected posts in the Batch Review Thread, ALL back-up judges MUST submit their judge notes, if required, to Nocturne or Krunkykai22 via PM (private message). Afterwards they will incorporate your notes to the Back-Up Judges section of the Batch Review Thread.


Now, onto the Judging Process. We have taken a LOT of time to go over what exactly needs to be judged and how it should be judged respectively. We have split the process into 6 distinct categories which we will go over thoroughly.

These six categories are:
  • Sync & BPM
  • Musical Composition
  • Technique/Interpretation
  • Engagement/Enjoyment
  • Flow
  • Overall Judges Score

Sync & BPM – When judging a file, the number 1 thing you should look for is how well the chart is synced to the song itself. Judges will look thoroughly to make sure sync is perfect, as well as are there any mis-rhythms or Beat Zero problems. They will also check to see whether the BPM (Beats-Per-Minute) is analytically correct, and make sure BPM changes are made respectively throughout the chart if needed. A very simple and straight-forward concept that is a must in any file be near perfect in order to fully play a file without obstacles.

Musical Composition – Judges will also be judging a file based on the music chosen, as well. This means, the structure of the song. In simplistic terms judges will be looking for:
  • Length of the song (Which MUST be over 0:45 in length for first time step artists unless noted otherwise)
  • If the song sounds linear without notes being given.
  • If the song drags or is repetitive for a constant amount of time.
  • And finally the quality of the MP3 chosen. (Bad quality, white noise, etc. etc.)

Technique/Interpretation – Here judges will look for how well the chart is stepped. Including, but not limited to, Color Theories, Inverse Theories, Gimmick Placements/Usage, Creativity, Grace notes, etc. As well as judging how well you interpreted the sound of the song to the notes you've chosen.

Engagement/Enjoyment – In this category, a judge will rate your file based upon how well the chart is enjoyable. Does the chart drag on? If it does, that could cost some valuable points. Judges will look for a chart to catch their attention and continuously provide reason to keep playing the file.

Flow – For our last category, judges will look over a chart to see how well the song flows from start to end. Does the structure of the file flow to the sound of the music? (pun) Does the structure of the chart have transitions that flow correctly? Lastly, is there correct scaling and progression? A perfect example of flow would be "Cirno's Perfect Math Class" as the chart itself provides a very good example of what we mean by Flow.

Overall Judges Score - For this final category, Judges will take the entire chart, and give it a score. This score is dependant on how well the Judges feel the chart was stepped. Not looking at any other category, a Judge will play through you file multiple times, then provide a score for how well they believe the chart was created.

Those are our categories for judging. In this next section we will explain the point value’s for EACH category while judging.

2 Categories will NOT have a point value, but that does NOT mean they are not worth anything. These 2 sections are:
  • Sync & BPM
  • Musical Composition

They are currently worth no value, but are an ABSOLUTE value that must be addressed and done well before even heading into the next 3 categories.
  • Technique/Interpretation – Will be graded on a (0-10) Scale, 0 being the lowest, and 10 being the highest.
  • Engagement/Enjoyment – Will be graded on a (0-10) Scale, 0 being the lowest, and 10 being the highest.
  • Flow – Will be graded on a (0-10) Scale, 0 being the lowest, and 10 being the highest.
  • Overall Judges Score - Will be graded on a (0-10) Scale, 0 being the lowest, and 10 being the highest.

Each judge will give 5 scores per file. The scores are as follows:
  • 1 Score for Technique/Interpretation
  • 1 Score for Engagement/Enjoyment
  • 1 Score for Flow
  • 1 Score for Overall Chart Score
  • 1 Score that Averages out the 3 categories above Overall Chart Score.

Once all that is said and done, the final step to finishing this system is to take the point values, and average them out to see if a file is good to go in game. The way we are going to do this is take all of the judges Overall Chart Scores add them up and take the Average. If it passes the threshold of minimum points needed to pass, then the file will be considered as Accepted.

The point value that is needed to pass a file is anything HIGHER than a 6.9. A 6.9 does not meet the requirement, however a 6.91, 6.92, etc. will be considered passing.

Once all the scores are tallied up, you will have 2 chances to get a song into game. The Overall Chart Scores mean will be one way; this Overall Chart Score will factor in how a judge grades the file from 0-10 on a subjective scale of fun, enjoyment, and interpretation. Another way is to add up each of the 3 categories and tally up what the average is and determine if it’s above the threshold. To show you a better example:

The 5 Judges have reviewed “Cirno’s Perfect Math Class” The order of scores goes:

Technique-Layering/consistency-Pitch Relevancy-Overall Chart Score-Average of beginning 3 categories
  • Judge A – 8/7/8 – 8 – 7.67
  • Judge B – 7/6/6 – 7 -6.33
  • Judge C – 6/7/7 – 6 – 6.67
  • Judge D – 6/5/7 – 6 – 6
  • Judge E – 7/8/7 – 8 – 7.33
Taking these values we would add up the Overall Chart Scores:
  • Judge A – 8
  • Judge B – 7
  • Judge C – 6
  • Judge D – 6
  • Judge E – 8
Total -> 35 -> Average = 7 (This IS PASSABLE)

The next values we would take are the averages of the 3 categories:
  • Judge A – 7.67
  • Judge B – 6.33
  • Judge C – 6.67
  • Judge D – 6
  • Judge E – 7.33
Total -> 34 -> Average = 6.8 (This is NOT PASSABLE)

These are the 2 formulas that we will be using for the duration of this system, until another formula is added, if needed.

Judge Notes

In this section we want to explain a little something regarding notes from the judges. At the top of this page, we mentioned an issue regarding “one-line” acceptances from judges on this site. We will not allow that to happen in this system. A judge MUST release notes for each category that has a point value on it. For example:

Something not acceptable:

Cirno’s Perfect Math Class:
File is awesome, put it in game. Accepted 9/10.

(This is not acceptable because there are absolutely no notes regarding any aspect of the file.)

Something acceptable:

Cirno’s Perfect Math Class:
This file shows an extraordinary effort in how well a chart can flow to a song. The way you stepped the chorus line, and the grace notes used were used in a way to not only emphasize the songs structure, but to show how enjoyable a chart can be with proper layering.

Now yes, we don’t expect everyone to be as much as a fairy as I was in creating that example (krunkykai22), BUT we do want there to be a very in depth description of notes. If the file has a lot of negative factors, list them. Don’t go all out to the point where if it’s the same mistake over a dozen times, you don’t need to list EVERY measure where it happens. You can GENERALIZE it in that fashion. But, please be descriptive at the least. Now, if a file has almost no errors and is perfect to you, then list the good points of the file. We don’t want one liners period. We might have first time step artists that won’t understand much, but they know that a file should be stepped accordingly. It is our job as JUDGES to explain with reason why we gave our notes. Help others become better step artists. It is why you are being chosen as a Judge on ThirdStyle to begin with.

Another thing we want to mention is what the perks to Judging would be. If you become a ThirdStyle Judge, we, of course will post up a new title that will give you the “ThirdStyle Judge” color. Not only that, but there will be *cough* a “reward” other than colors. This reward will be a salary!

That’s right, ThirdStyle will PAY YOU to be a judge here. (Before you jump for joy, it won’t be in US DOLLARS! It will be in Stylepoints Side Note:

All Judges will not receive a salary right away. You must maintain a "probation period" before you can be eligible for this reward. The "probation period" is a 2 month period. (60 Days - with a minimum of 4 batches judged.) After this 2 month period, the system advisors will overlook your work, and then promote you out of "probation." This "period of time" is only in place to make sure that people don't just "half-way" judge a file looking for the SP reward. We want to make sure that ALL Judges give their all when they come to Thirdstyle.

RULES

Okay everyone, we all knew this was coming. So, along with our system and threads, we will have a basic rule system that will express a ZERO TOLERANCE in any disrespect and foul language being used by anyone against anyone else. So here are the rules:
  • No Flaming
  • Demeaning Notes by judges
  • Overall Lack of Activity by the Judges
  • All acquired Judges will be required to send in their notes ON TIME and must be sufficient in a professional manner or consequences will happen (ie, removal, replacement)
  • In any of the threads there should not be any arguments or petty disagreements. If you have an issue with your file review, either ask politely in the Discussion thread or bring it to a PM.
  • We won’t tolerate anyone “putting down” anyone else based on their step-chart or file review and we will push consequences extremely fast to resolve this issue ASAP.

Contrary to our rules, we do want everyone to joke around and have fun, but only at the expense of fun. NOT at the expense of anyone else.

Alright, everyone that is the system! It may not seem like the “golden child” of ThirdStyle, but we worked diligently at making this work. We believe that this system will bring together a higher morale on ThirdStyle that will peak the traffic for files to flow through. We know everyone wants more files in game. We know people have submitted files and waited close to a month for the file to be reviewed. This will settle all of those problems and situations. Give it time and it will work.

If you have any questions or concerns, please feel free to PM the following people who were included in creating this OP:
Nocturne
Krunkykai22
Ichliebekase
Vote4Nixon

Selected Judges:

Judge 1.) Midigami (Jesus)
Judge 2.) Nocturne (Shaun)
Judge 3.) Ichliebekase (Sara)
Judge 4.) Krunkykai22 (Frankie)

Back-Up Judge 1.) eiddiM (Austin)
Back-Up Judge 2.) DefTeaMachine (Christian)
Back-Up Judge 3.) Housekeeper (Ian)
Back-Up Judge 4.) yoshl (Chris)
«1

Comments

  • samurai7694samurai7694 Simfiler
    edited February 2013
    "I wish to be a TS Judge"

    I'm getting more free time since school work isn't really that big of a deal for me anymore.
  • MidigamiMidigami Official Main Simfile Judge
    edited February 2013
    I wish to be a TS Judge. I'll do whatever it takes to get my notes in on time, even if it means using the college comps in the library during the week xD
  • FusionFusion Moderator
    edited February 2013
    I PM'd you Luis, and Miid, I'll be talking to you on Skype :)
  • PowerstarsPowerstars Senior Member
    edited February 2013
    In any case, maybe now we can get files judged faster.
  • FusionFusion Moderator
    edited February 2013
    That's the whole point of this system. We want to get as many files stepped and sent in to the batch as soon as possible. We will release a new Batch E-Mail once we discuss it with Tiza.

    Having issues editing the OP, will talk to tiza about it in the morning:

    BUT:

    Welcome our FIRST Official Thirdstyle Main Judge - Samurai7694 (Luis)!!!!!!

    EDIT - made an update to the bottom of the OP with Luis' selection.

    EDIT 2:

    Welcome our SECOND Official Thirdstyle Main Judge - Midigami (Jesus)!!!!!!
  • eiddiMeiddiM Official Back-Up Simfile Judge
    edited February 2013
    I hope to see this system go well.

    Also to be clear, I am not interested in being a judge by posting in the thread. (I don't want any confusion.)

    As a side note, if you need to give an applicant a chart to review as a test of their reviewing talent, please feel free to use The Dream of Scarlet as a chart for that test.
  • FusionFusion Moderator
    edited February 2013
    That's pretty ironic, considering your file is going in as first to be judged in the new system.

    That's sort of why I kind of said to sit tight. Don't worry, Austin, you're file will get reviewed lol
  • PowerstarsPowerstars Senior Member
    edited February 2013
    I would like to be a back-up judge. No Skype, so you'd have to interview me through PM.
  • HousekeeperHousekeeper StepCloud Mod
    edited February 2013
    yo guys can someone start up a quick skype group with everyone so we can discuss this and stuff
  • FusionFusion Moderator
    edited February 2013
    On a side note, we already have 6 files that will be added to the first batch we get once this system is complete with Judges and back up Judges. :) So everyone is encouraged to step as many files as possible so we have a grand opening so to speak.

    EDIT: Ian get on skype.
  • MidigamiMidigami Official Main Simfile Judge
    edited February 2013
    I present Viking fjord to erry1
  • FusionFusion Moderator
    edited February 2013
    Alright, done for the night everyone. Go ahead and keep sign-ups coming. Tomorrow members of this system will speak 1 on 1 with you regarding your knowledge and experience with Judging/Simfiling. We really want this system to give a little boost to morale in the community and provide more entertainment in the form of Charts so people keep coming back. As well as obtaining more members to the community. :)
  • NocturneNocturne Simfiler and <span style="font-style:italic">Freakin Graphics Master</span>
    edited February 2013
    You know I'm in on this for main judge man. I'll do the batch zipping and sending them to each main judge each batch. Can't wait!
  • FusionFusion Moderator
    edited February 2013
    Added Nocturne (Shaun) to the OP for Main Judge 3 :)

    Congrats man :D
  • MrTeaMrTea Official Back-Up Simfile Judge
    edited February 2013
    A salary. Welp I'm interested now.
  • cheezpuffscheezpuffs Senior Member
    edited February 2013
    Cool, but kinda confusing. .-.

    I sent my MEOW simfile yesterday, now what?
  • DrugstoreCowboyDrugstoreCowboy Administrator
    edited February 2013
    I read the entire batch section by replacing batch with bitch, batches with bitches. Made for an excellent read.
  • MrTeaMrTea Official Back-Up Simfile Judge
    edited February 2013
    DrugstoreCowboy;36101 said:
    I read the entire batch section by replacing batch with bitch, batches with bitches. Made for an excellent read.
    Thanks for the idea. Im now even more interested.
  • FusionFusion Moderator
    edited February 2013
    LOL Drugstore, that's hilarious. I actually went and read it like that ... lulz

    Anyway, Deftea, if you are really interested PM me and i'll talk to you about everything.

    Cheezpuffs - If you have any questions, PM Me as well and I can go over what is confusing you. :)
  • FusionFusion Moderator
    edited February 2013
    We have our first Back-Up Judge!!!

    Welcome eiddiM (Austin) as our FIRST Official Back-Up Judge!
  • eiddiMeiddiM Official Back-Up Simfile Judge
    edited February 2013
    krunkykai22;36125 said:
    We have our first Back-Up Judge!!!

    Welcome eiddiM (Austin) as our FIRST Official Back-Up Judge!
    image

    I did it?
  • XandertraxXandertrax Simfiler
    edited February 2013
    Oh god. TS switches to batch format. Baaaaaaaaaaaaaad news.

    p.s. I would judge but I can't really judge the technicality of hard files. Maybe back-up?
  • MrTeaMrTea Official Back-Up Simfile Judge
    edited February 2013
    to be honest, I'm not sure if batch format is the best idea.
    Im more supportive of something a little more free flow.
  • NocturneNocturne Simfiler and <span style="font-style:italic">Freakin Graphics Master</span>
    edited February 2013
    Xandertrax;36246 said:
    Oh god. TS switches to batch format. Baaaaaaaaaaaaaad news.
    Whatever gets the ball rollin I suppose. Better than what we have now.
    DefTeaMachine;36250 said:
    to be honest, I'm not sure if batch format is the best idea.
    Im more supportive of something a little more free flow.
    We were completely free flow, suffered one month of no reviews or releases. How is that in any way more productive than a batch? lmao

    Our simfilers lack discipline and the ones dedicated to helping the batches deserve a reward for their work, not for sitting around doing what they want.
  • FusionFusion Moderator
    edited February 2013
    I agree with Shaun 100%. We spent hours fixing out all the details on this system. We know that with a 10 man group of main and back up judges that we WILL get productivity going with the flow of files on this site. All we need is a little push to go ahead and get things done.

    Xander, yes you can back-up judge if you want, but just PM me so we can discuss the details and go over a little "interview" if you do not mind so I can explain everything (:

    EDIT: And Batches, just so you guys know, aren't a "bad idea." More or less, from what you guys have seen on FFR, are quite lengthy and really unorganized with drama and what not. Here, we have went over every aspect of it, and WILL BE enforcing rules to the upmost respect. This WILL be organized, and it will flow really well as long as the Judges cooperate fully enough. So don't worry, have faith in us, we didn't do this knowing the pros and cons.
  • ZeroZero Simfiler
    edited February 2013
    Fixing the judging format to get the judges more active and all is great, but the problem still remains that no one is really stepping files. Even if you are able to get through files at 4x the speed, that won't mean much if there are no submissions. You now have to find a way to motivate people to step, that is just as great as a problem as the judging process. I do not think there is any quick fix way to get peoples motivation up, you cannot really just tell them to step, as that obviously hasn't worked before. Good job with the new judging format, onto the next problem!
  • FusionFusion Moderator
    edited February 2013
    Don't worry about that, we also have ideas for stepping and what not. We have been doing everything we can, BUT it's not JUST UP TO US!

    EVERYONE, needs to do their part for this website. Not just staff, but members and such. They have to WANT to keep the site going. This is just one way for us to push in a direction forward. Don't criticize what we are doing in any way because quite frankly, there isn't anyone else that's come forth with ideas and suggestions.

    In the past month (4 weeks) i've been working on 15 different things at once. So has Shaun, so has Sara, and so has Tiza. We are doing everything we can, but we're just 4 people. 4 people to push ideas and suggestions forward for a lot of people that are here. We can't afford AFK'ers. Disappearances, and the sort. Anyone that wants to help, can help at their own pace and regard, but we want EVERYONE to help out. A LOT of things are going to happen over the spring and summer, and we want the biggest amount of members to sign in and work with it. We're doing our best to increase traffic flow, along with ads. There's a bunch.

    So, basically, stick around and help out. The rewards will definitely be greater then the work.
  • eiddiMeiddiM Official Back-Up Simfile Judge
    edited February 2013
    Zero;36265 said:
    Fixing the judging format to get the judges more active and all is great, but the problem still remains that no one is really stepping files. Even if you are able to get through files at 4x the speed, that won't mean much if there are no submissions. You now have to find a way to motivate people to step, that is just as great as a problem as the judging process. I do not think there is any quick fix way to get peoples motivation up, you cannot really just tell them to step, as that obviously hasn't worked before. Good job with the new judging format, onto the next problem!
    One could argue a method of doing so is to bring about a more constant system that honestly, more people are more than likely used to. Assuming the batches happen at a reliable rate, I fail to see how this wouldn't encourage a stepper to say "hey my chart is actually going to be judged in a fair amount of time, versus what was happening and that in some cases it took a month plus." It gets good reviews to the users faster and I think that's the primary goal, to start off with encouraging people with reliable notes at a reliable rate.

    Who is to say more may not come later? I surely don't know the answer here.
  • FusionFusion Moderator
    edited February 2013
    Good reply Austin. That was our first motivation for even beginning to do this system. We know that most people do not want to step here because it would take weeks or even months to be reviewed and put in game. So our motivation first hand was to get a stable system involved. THEN, see if it pushed forth the flow of files. Now I can tell you that most of us are going to be stepping songs very soon once this system is complete. Back-Up judges will definitely have their share for the first batch or second because some of the main judges will probably step files along with me and Shaun and Sara.
  • Vote4NixonVote4Nixon BADministrator
    edited February 2013
    DefTeaMachine;36250 said:
    to be honest, I'm not sure if batch format is the best idea.
    Im more supportive of something a little more free flow.
    There are ups and downs to every system, and yeah, I enjoy having something more free flow as well. It allows simfilers to operate more autonomously, but there is the responsibility of actually getting things done, which is something that isn't intrinsic to the simfiler title. Normally, a simfiler is considered a content contributor as opposed to a content manager, but given the original judging system, the title had a dichotomy of both roles. It worked for a while and improved the simfile process when TS was first launched, that is, having Tass distribute Rickycore and other mp3s and throwing in whatever files he got. But, as the site and user base progresses, change is bound to happen, and this is a necessary step in evolving our current methodology so we can get back to releasing 2-3 files a week.
    Zero;36265 said:
    Fixing the judging format to get the judges more active and all is great, but the problem still remains that no one is really stepping files. Even if you are able to get through files at 4x the speed, that won't mean much if there are no submissions. You now have to find a way to motivate people to step, that is just as great as a problem as the judging process. I do not think there is any quick fix way to get peoples motivation up, you cannot really just tell them to step, as that obviously hasn't worked before. Good job with the new judging format, onto the next problem!
    YOU + ME. ONE TIME. Haha, this is true. One hope is that if simfilers know that their file won't be sitting for months without a review, they would be inclined to step more. Another hope is that this demonstrates to not only simfilers, but also the community as a whole that TS is not dying away anytime soon and will remain a vibrant, fun place for people to put their time and effort into. From my experience, simfiles tend to come in waves from a specific simfile author. There's always a drought followed by a surge of motivation resulting in the next wave of files. We just need enough active simfilers so that when some are on a drought, others are on a surge.
Sign In or Register to comment.