iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
In a first-person account published on DEV Community on September 30, 2026, developer Tushar Agarwal describes building Quorum, a self-hosted hackathon platform. Its most striking finding came before he designed any results screen. In the DOGFOOD 2026 fixture, judges agreed no better than chance: the intraclass correlation was -0.006, and a permutation test with 2,000 shuffles returned p = 0.504.
That measurement anchors the lessons below. A ranking can look definitive when the scores beneath it do not support it, and the choices made before scoring, including how judges are assigned, how ties are handled and how access is enforced, decide how much trust a final list deserves. Every figure in this article is the author’s own report from one fixture. None has been independently reproduced, and the fixture should not be read as evidence about hackathon judging in general.
What the agreement test measured
The fixture contained 41 submissions (one of them a duplicate), 30 judges, 123 reviews and eight tracks. Before building the results experience, the author measured how consistently judges scored the same projects.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The statistic was ICC(1), a one-way intraclass correlation. It asks how much of the variation in scores is explained by which project was being judged, as opposed to scatter among the judges who reviewed it. The author’s value of -0.006 means project identity explained essentially none of the variation. A value this close to zero is consistent with no shared signal at all. It does not show that judges systematically disagree.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
The permutation test shuffled the scores 2,000 times to see how often chance alone produces agreement at least as strong as the observed agreement. About half of the shuffles did (p = 0.504). The author’s conclusion is that judges agreed no better than chance in this fixture. Whether a different event would show the same pattern can only be answered from that event’s own data, and the lessons that follow are about what to do with that uncertainty.
Why a ranking can look definitive when the data is not
A list sorted to two decimal places looks like a finding even when the data beneath it cannot separate neighboring projects. The author’s central point is that a leaderboard hides its uncertainty unless the platform is built to show it.
His prescription has three parts. The platform should show uncertainty around each position, not only the order. Tie procedures should be defined before scoring closes, not improvised when two projects are level. When an organizer resolves a close call, that decision should be recorded in the audit trail rather than folded silently into the final list.
Per-judge z-scores: normalization that can backfire
Many organizers correct for harsh and lenient judges by converting each judge’s scores to z-scores. The author found that this method breaks down in ways that matter for a small panel.
Where z-scores fail in the fixture
- One judge gave every project the same score, so the standard deviation was zero and the z-scores were undefined.
- Two judges had only one review each, which leaves too little data to estimate their spread.
- When a judge has few reviews, z-scores can erase real differences between projects.
- Judge severity becomes confused with the quality of the batch that judge happened to receive.
What the simulation showed
The author built a simulation based on the fixture’s assignment design and compared two methods at a moderate spread of judge leniency. The table shows the share of runs in which each method selected the true winner.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
| Scoring method | True-winner selection, moderate leniency spread (author’s simulation) |
|---|---|
| Per-judge z-scores | 23% |
| Raw average | 29% |
In that simulation, normalization made the top-line result worse, not better.
The replacement is a random-effects model. Each judge receives a leniency offset, and those offsets are shrunk toward zero so that a judge with few reviews cannot move projects much on the strength of a handful of scores. The amount of shrinkage is estimated with REML (restricted maximum likelihood). The author reports Kendall tau improvements, a measure of how closely a full ranking matches the true order, for two leniency spreads:
| Judge leniency spread (simulation) | Reported Kendall tau improvement, random-effects model |
|---|---|
| 0.4 | +0.018 |
| 0.8 | +0.070 |
The improvement is larger at the wider spread, which is where judges differ most and where calibration has the most to correct.
Assignment design sets the limit on calibration
The author’s most transferable point is that calibration cannot recover information the assignment never collected. If one panel always reviews one group of projects, the judge’s leniency and the group’s quality become entangled. Normalization cannot separate them, because every judge’s average is pulled by the same projects.
| Assignment pattern (author’s simulation) | Reported gain from calibration |
|---|---|
| Separate panels, each reviewing its own project batch | Zero |
| Overlapping batches, with judges sharing projects | +0.036 |
The author sums this up in one line: “Your assignment algorithm is part of your scoring algorithm.” Quorum’s planner therefore aims to maximize overlap between judges and projects, and it displays judge connectivity so organizers can see whether their judging network is linked at all. A panel that never shares a project with another panel is a warning sign, however sophisticated the scoring math.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Spend extra review where prizes are decided
The first version of the focus-round planner considered only the overall podium. The fixture, however, also had a Best in Track prize in every track, and those contests were just as exposed to close scores. The revised planner directs extra review effort to uncertainty around either the overall prize or a track prize.
Recommended Free Tools
When a boundary stays unresolved, the author describes this sequence:
- The head-to-head round is defined in advance, before scores are in, so its format is settled before anyone knows which projects it will affect.
- Three judges with no conflict of interest with either contending project take part in the round.
- If the round still does not resolve the order, the organizer makes a decision in writing, and that decision is logged.
Judge isolation and the invariants the database enforces
The author calls judge isolation the security core of a judging platform: “a judge must never see another judge’s scores.” Isolation is enforced in layers. Explicit route policies determine which endpoints a request may reach, and score access goes through scoped repository functions rather than ad hoc queries. Beneath the application, database triggers enforce rules that application code should never be able to bypass:
- The audit log is append-only, and each entry is hash-chained to the one before it, so altering a past entry breaks the chain.
- Ranking runs are immutable once created.
- Scores must fall within defined bounds.
- Scores are frozen when a deadline passes.
The author also explains why he did not rely on row-level security (RLS) for judge isolation. His concern is that RLS can silently filter rows out of a query. Cross-judge calculations such as normalization or aggregation would then run on incomplete data and return plausible but wrong numbers. This is his rationale for his own architecture, not a general rule against RLS.
How to check that isolation holds
The author asks the question directly: “How do I know isolation holds?” His answer has three parts. First, a written authorization matrix that lists which role may read which data. Second, a canary crawl, which checks whether marked test values surface anywhere they should not. Third, a live cross-role probe that attempts each role’s access against a running instance. The author reports 344 cross-role probes with zero leaks, and 188 tests run against a real Postgres database.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Refusals are part of the interface
The author’s test advice is “test what a refusal says, not only its status code.” A request can return the correct 403 or 404 and still leak information through its message. If a refusal reads differently for a real judge ID than for a made-up one, it tells an attacker which IDs exist.
For late submissions, the order of checks matters:
- Authenticate the caller.
- Check the caller’s role.
- Check the deadline.
- Validate the input.
In this order, a late submission from a valid user receives the deadline refusal rather than an error about its contents. Refusals should never reveal whether a judge ID exists.
Offline claims need an offline test
Quorum is designed to run without internet access, and the author’s check for that claim is built around the network itself. The check runs in a Docker network marked internal. Before anything else, it verifies that the network cannot reach the internet. Only then does it run the checkers and the isolation probe, so a passing run means the tests executed in a network that was actually cut off.
The local AI model ships inside the image, which is what makes offline operation possible. The author reports checker results of 7/7 and 21/21.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsContainer limits, proxies and migrations
Size workers to the container, not the host
Python’s os.cpu_count() reflected the host machine’s capacity rather than the limits of the deployment container. The author’s concern was that a worker count derived from it could exceed what a 1 GB memory limit can hold. He reports that Quorum’s workers use about 85 MB while idle and about 170 MB with the local model loaded. The lesson is to derive worker counts from what the container allows, and to measure memory with the model loaded, not at idle.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Trust the right proxy hop
For proxied deployments, the author describes two safeguards. Metrics pages are guarded whenever forwarding headers are present, so they are not exposed through a proxied request. Client addresses are read from the rightmost forwarded hop that is not the platform’s own proxy. A client can set the leftmost value, so trusting it would let a caller spoof their address.
Serialize migrations across replicas
When several replicas start at once, each could try to apply the same schema migration. The author uses a Postgres advisory lock so that only one replica runs migrations at a time. In a load test of 300 voters across 3 replicas, the author reports zero lost votes.
Keeping the AI on a short leash
Quorum includes an assistant, and the author’s design rule is that it selects tools rather than generating facts or scores. A 23 MB sentence-embedding model runs locally on CPU and matches a request to one of 15 named skills. Skills run permission-checked queries, so the assistant’s reach is limited to what the caller is allowed to see.
Some questions are deliberately unanswered. The article says a judge who asks who is winning receives no answer. A feedback coach reviews written comments for concrete next steps and for coverage of the scoring criteria, but it never assigns a score.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
