Comments (18)

UK Video Forensics Insights on Jan 23, 2014
David Spreadborough has been a police officer in the UK for 23 years, a forensic video investigat...

Beaten Nearly to Death, Weak Video Lets Attackers Go Free on Jan 22, 2014
Watch this: If you were a juror could you make a decision on what happened here? Eyewitness te...

Cruise Ship Surveillance Reviewed on Jan 21, 2014
Falling overboard to die alone in a dark cold sea is terrifying. Video analytics might solve this...

Store Employee Explains How They Stole on Jan 20, 2014
In 2004, Lauren started work at a large CVS in a Washington, D.C. suburb. She says she really nee...

Silicon Valley Startup Targets Neighborhood Surveillance on Jan 16, 2014
Silicon Valley is warming up to the surveillance market, seeing the potential of consumers to emb...

ALPR / NPR Recognition Accuracy and Issues on Jan 15, 2014
Sick of red light and speed cameras? A lot of people are. There are entire websites and web forum...

Cable Company Home Security Issues on Jan 14, 2014
Mark already had cable from Comcast so when he started getting offers for their home security ser...

Criminal Explains How Cameras Caught Him on Jan 13, 2014
Henry’s story is the second in our series exploring surveillance from the other side of the...

Guard Fired For Reporting Surveillance Sexual Harassment on Jan 10, 2014
A guard is fired after reporting misconduct by another guard monitoring surveillance cameras. Now...

Most Recent Industry Updates

LAST Chance - IP Camera Course Winter 2014 on Jan 25, 2014
This is the last chance to get into IPVM's IP camera course that starts next Tuesday, January 28t...

UK Video Forensics Insights on Jan 23, 2014
David Spreadborough has been a police officer in the UK for 23 years, a forensic video investigat...

Ask the AHJ: Chicago on Jan 23, 2014
The AHJ has the final say in access control systems. That power, combined with the rules not alwa...

Cloud VMS For $5 Per Camera Per Month on Jan 22, 2014
A startup launched by the founder of Baracuda, a networking company worth more than a billion dol...

Beaten Nearly to Death, Weak Video Lets Attackers Go Free on Jan 22, 2014
Watch this: If you were a juror could you make a decision on what happened here? Eyewitness te...

Cruise Ship Surveillance Reviewed on Jan 21, 2014
Falling overboard to die alone in a dark cold sea is terrifying. Video analytics might solve this...

Delayed Egress Maglock Claims to Save Thousands on Jan 21, 2014
Deploying delayed egress right is tricky even when codes allow it, and making sure you meet all t...

Exacq Breaks Avigilon Integration on Jan 20, 2014
IPVM testing has verified that Exacq has broken integration with Avigilon cameras. Up until mid y...

Store Employee Explains How They Stole on Jan 20, 2014
In 2004, Lauren started work at a large CVS in a Washington, D.C. suburb. She says she really nee...

As I was getting ready to leave the office today, I started to google for a movie to go to with my wife tonight. I’d been testing sites in chrome and firefox all day, so without thinking I typed my search term into the firefox address bar. What I got back astounded me:

Yep. Instead of responding with NXDOMAIN as a good DNS resolver should, it redirected me to their ultra-spammy “search page.” I’m running debian, so the chances of there being a virus are pretty low, and some research confirmed my suspicions:

jbert@vps:~$ dig cinetopia
; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> cinetopia
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14427
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;cinetopia. IN A
;; ANSWER SECTION:
cinetopia. 10 IN A 198.105.254.11
;; Query time: 2 msec
;; SERVER: 4.2.2.3#53(4.2.2.3)
;; WHEN: Fri Jan 24 22:11:03 2014
;; MSG SIZE rcvd: 43
jbert@vps:~$ cat /etc/resolv.conf
nameserver 4.2.2.3

This has a few ramifications. First, I don’t blame them for wanting to monetize these servers. Everyone and their brother uses them for pings, DNS resolution, etc. — servers aren’t free and bandwidth isn’t free. So I get it. But, for one, I have no idea what else this page might or might not be doing. Is it setting third party tracking cookies? Tracking/bubbling my browser fingerprint? At the least it’s leaking, in clear text on the wire, things that I expected to be sent to an encypted DDG search. If there was sensitive search terms or information in that query, it just dropped into Level3′s logfiles. Additionally, I have a few scripts that rely on a nonexistent domain getting an NXDOMAIN response. They’ll now break.

So, be warned. One can no longer rely on 4.2.2.x DNS for RFC-compliant responses.

EDIT: On further testing, this does not appear to be happening from other locations. It’s also stopped happening from this VPS. It’s worth noting that this is not the first time I’ve noticed this, just the first time I’ve had the chance to grab some evidence and make a post about it.

EDIT 2: Looks like this isn’t the first time this has popped up. Here’s a link to a vpsboard.com thread from 11/2013 from a user who noticed the same thing: https://vpsboard.com/topic/2542-level3-public-dns-servers-search-engine-redirect/

“I’m Rachel,” I say to the man who is here to evaluate me, extending my hand, trying to put on my best sane face. Problem is, no one ever told me what that looks like.

He eyes me for a moment, then takes my hand.

I run him through the story, trying to emphasize my efforts to be honest and to get help.

I say, “So as soon as I cut, I texted my FroCo for support.”

“But you admit that you willfully harmed yourself?” he says, like he’s just won something.

“Well … yes.” Because obviously I admit it. I’m not a liar. If I were a liar, I would never have gotten myself into this mess. Fuck me for not being a liar.

And so, when I say “yes” to the ‘I admit cutting myself’ part, he nods his head and closes his eyes like someone has just given him a bonbon.

I tell him when I come back to Yale, I will get a therapist on campus and keep working with the one I have at home. I will stop cutting.

“Well the question may not be what will you do at Yale, but if you are returning to Yale. It may well be safer for you to go home. We’re not so concerned about your studies as we are your safety,” he says.

“I’m sorry,” I say. “What makes you think I will be safer away from school, away from my support system?” School was my stimulation, my passion and my reason for getting up in the morning.

“Well the truth is,” he says, “we don’t necessarily think you’ll be safer at home. But we just can’t have you here.”

* * *

On the night of Jan. 27, 2013, I slashed open my right thigh six times with a Swiss Army knife. I then spent four hours thinking about how good it would feel to jump off the fifth floor of Vanderbilt Hall. On Jan. 28, I put on a pretty dress and went to class. Before lunch, my cuts had stained it brown.

That night I texted my Freshman Counselor to tell her what had happened, just as I had done all the other times I felt suicidal and had cut myself. When I went to her suite, I showed her the gashes.

We went to Yale Health Urgent Care, at around 11:00 p.m., where a doctor bandaged my leg. A psychiatrist appeared. I told her that I had experienced suicidal thoughts the night before, but that the cuts had not been a suicide attempt. I told them that I was no longer suicidal.

At midnight, I was strapped to a stretcher under the ashen ceiling of an ambulance, on my way to Yale-New Haven Hospital. There I was taken to the locked ward of the ER — guarded by officers with guns — stripped of all my belongings, including my pants (they had a drawstring), and shunted into a cubicle containing nothing but a bed. I was here for my own good, they told me.

For 24 hours I had nothing to do but listen to the rattling gasping sound coming from the person two beds down, and to a schizophrenic person declare, every hour or so, that he had soiled himself. I was asked to recite the presidents of the United States, in reverse order, as part of a psychiatric evaluation. For more than a day I was not permitted to make a phone call. For more than a day no one had any idea where I was — not even my parents.

When a bed opened up in the actual Yale-New Haven Psychiatric Hospital, I was transported, again in an ambulance, and introduced to the place I would spend the next week of my life. Upon arrival, I was taken into a small room with two female staff members, forced to take off my underwear, spread my legs, then hop up and down to make sure nothing was hidden “up there.”

My Freshman Counselor had brought me some extra clothes and a course packet for my travel writing class, so that I would have something to read. The course packet was confiscated. Why? Because I might cut myself with the plastic binding — you know — the kind you get from Tyco. I might commit suicide with that, they said. “You’re a cutter,” they told me.

For a week, I was not allowed to set foot outside. I was not allowed to stretch my hamstrings or calves or any other body part. I was not allowed to pace my confines. I was not allowed to drink caffeine. I was not permitted to take ibuprofen for my caffeine withdrawal headache. I did not get to take a shower until my third day. Phone usage was restricted and phone calls were closely monitored. I was threatened, by a nurse, with the possibility of having my wrists and ankles tied to my bed, and witnessed this threat be carried out on others. Whoever built the hospital had termed this ward, “Liberty Village.”

There was little “treatment” in the hospital. Mostly, we watched television, played Pictionary and Connect Four and sat. I was interviewed by various clinicians a few times a day; I saw my assigned psychiatrist only three times, for half an hour or so, over the course of seven days. This limited treatment was fairly standard for all patients, but it soon became clear that it would have little effect on my situation.

The milieu counselors, nurses and doctors in Yale-New Haven Psychiatric Hospital have absolutely no input when it comes to deciding who gets to stay at Yale and who is forced to leave. In talking to the nurses, doctors and fellow Yale students I encountered in the hospital, I understood that job to belong to Dr. Eric Millman and to chief of Yale Psychiatry, Dr. Lorraine Siggins — two people who work for the University, rather than the hospital.

I have shared with you my memorable exchange with a senior psychiatrist at Yale Mental Health who came to evaluate me. It was this exchange that led me to keep an extensive and thorough journal during my time in the hospital.

But Dr. Siggins is the one who makes a ruling: Does Johnny stay at Yale or does he go? And in my talks, a consensus emerged: Dr. Siggins does not always — and by some accounts, rarely — make contact with the student in question. (A Yale senior who was in the hospital with me was not granted a meeting with Dr. Siggins but was still forced to leave Yale.) Neither the staff members I spoke with nor a fellow Yalie who had prior experience in the hospital knew of any Yale student admitted to the hospital who had been allowed to stay at Yale.

My interview left me terrified of the possibility of leaving school. I called my parents, and they promptly put themselves on Dr. Siggins’ radar, meeting with her twice and securing me a personal interview. All I remember was that my mind was totally blank when I spoke to her, because I was so focused on making her believe that I was “okay.” This, of course, is totally futile when you’re sitting on a cot in a mental hospital.

She called me three days later to tell me that I would have to go home. That meant that I was forced to formally withdraw from the college, with no guarantee of return. As soon as her decision came down, I was eligible for release into my parents’ custody. Upon my release from the hospital (also not a function of my recovery — but as a result of my expulsion from the College I was even more depressed when I left than when I was admitted, my Yale ID was confiscated, as was my room key. I was given one evening to pack up my entire life.

My college dean told me I was not even allowed to spend the night in my room in Vanderbilt Hall. I fell asleep on the futon in my suite’s common room at four a.m., breaking the rules, but exhausted and unable to continue putting my things in boxes, dismantling the reality of my college life. I had a chance to say goodbye to a few friends — most of whom I would not hear from during my time away. 18 hours after I walked out of the hospital doors, I was on a plane, headed back to North Carolina in a storm of tears.

I did what they said was necessary to be a candidate for readmission: therapy, more therapy, two college courses, more therapy. And I healed. Mostly.

I filled out the paper application for readmission: the usual demographic crap, a three-page personal statement, a transcript of my summer classes, two letters of recommendation, a profile from a therapist and a check for $50. I flew to New Haven for my three interviews — with the dean of my residential college, Dean Pamela George (chair of readmission) and Dr. Siggins.

As a side note, I might mention that Dr. Siggins was 45 minutes late to my interview. Dean George called me an hour before the scheduled time to cancel, forcing me to interview the following day, two hours before my return flight took off. I answered every question with as much positivity as I could sell. I said: I do not cut, I do not think of killing myself. I am great. Two weeks later, I was readmitted.

Every morning of my year away from Yale, I woke to the sight of the “Yale” pennant on my bedroom wall — the one they send to accepted freshmen in the big, glorious “Welcome to Yale” packet. “You’re in!” it says. “You’re a treasured asset to our University!” it says. “Come to Bulldog Days and feel the love because we love you and we care about you and we don’t want you to go to any other school because you’re the shit!” it says.

Thinking back to that welcome packet, there is a conspicuous omission: *We love you and want you and will provide for you and protect you, as long as you don’t get sick.*

* * *

I return to a different Yale, though it is I who have changed. After a year spent focusing solely on my health and well-being, I find myself, though not perfectly balanced, resting closer to my ideal center. And, after a year of watching and analyzing every one of my inner ticks, I see external things that were invisible to me before.

I see that Yale is a fundamentally unhealthy place in one important way. The problem is, everyone is “okay.” I have known friends who have suffered the deaths of siblings, who have been victims of sexual assault or who have fought life-threatening illness, all while navigating their sexuality, while taking five-and-a-half credits, while chairing more organizations and running to more meetings than they can keep track of. I have known friends to do all of this and still profess, at every opportunity, to be “okay,” “fine,” “great.”

To say something else, to be — in our own minds and in the minds of others — something else, is for some reason not acceptable at Yale. None of us are completely okay. But the pressure to conform to being perfectly functional and happy is a burden that we should neither want nor bear.

Where does it come from? For most students at Yale, I think the pressure is subconscious, upheld through day-to-day conversation: My classes are amazing. My extracurriculars are dope. My internship this summer is baller. Life is awesome. Are you awesome? No one wants to deviate.

But I think the source is not, in fact, the students. Those of us who have admitted, at some point or another, that we are legitimately not okay, have learned that there are real and devastating consequences of telling the truth. Because Yale does not want people who are not okay. Yale does not want people who are struggling, who are fighting. Yale, out of concern for its own image, wants them to leave. And Yale makes them.

With this, I refuse to be okay.

20 hours ago

STOCKHOLM — Newly crowned Norwegian world chess champion Magnus Carlsen took just nine moves to checkmate Bill Gates in a speed game to be aired later on Friday.

Challenged to a game in a chat show hosted by well-known Norwegian television presenter Fredrik Skavlan and due to be shown in Norway, Denmark and Sweden, Microsoft founder Gates said before the game that the challenge had "a predetermined outcome."

Gates, 58, who was ranked by Forbes magazine this year as the world's second-richest person behind Mexico's Carlos Slim, had 2 minutes to make his moves against just 30 seconds for Carlsen. He lost to the 23-year-old in around 1 minute 20 seconds.

"Wow, that was fast," he said to Carlsen, whose rockstar appeal has won him the moniker the "Justin Bieber of chess."

The program was recorded on Wednesday in London, Norwegian TV NRK said.

Asked by Skavlan under what circumstances he felt intellectually inadequate, Gates answered: "When I play chess with him (Carlsen)."

Carlsen, a grandmaster since he was 13, received non-stop television coverage in Norway when he beat defending champion Viswanathan Anand of India last November to take his first world title.

Copyright 2014 Thomson Reuters.

David S. Peck is getting a lot of emails. In a glitch possibly related to the massive Gmail outage underway right now, there’s an odd bug in Google search which is pointing users directly to his personal email address. The address appears in a “Compose” window that pops up when the top search result for Gmail is clicked. Yes, it’s bizarre. Very, very bizarre.

Several of us at TechCrunch have been able to duplicate this bug, first brought to our attention by a tipster. Given whatever is going on with Gmail right now, your mileage, as they say, may vary. 

To reproduce the bug, first search for keyword “gmail” on Google. The top organic search result says “Gmail – Email from Google,” and beneath that are two other sub-links, one that says “Email” on the left, and the other which reads “Gmail – Google.” Click the one on the left (where the text reads “10+ GB of storage, less spam, and mobile access. Gmail is email …”)

A Gmail compose window appears, and the email address dsp559 at hotmail – which none of us have in our address books – is automatically filled in.

Another Google search surfaces who this address belongs to: one David S. Peck of Fresno, California. We even found his resume.

We spoke to Mr. Peck on the phone just now, and he says he’s been receiving thousands of emails, the majority of which are blank.

“I’ve been getting thousands of no-subject, blank emails,” he says. “500 of them come every hour, I can’t stop them.”

The issue actually started yesterday, he says, and he contacted Hotmail support this morning to try to get help. Ironically, he asked them to contact him at his alternate email address, which is Gmail.

Peck will not see their reply anytime soon, it seems.

The Hotmail address is his main account, and he’s now missing his important emails while under this deluge. “They’re coming so fast, I want to stop them. I deleted everything last night and woke up this morning and had 1,900 new emails” he says. “Only two of them were emails I cared about,” he says. 

Though most of the emails are blank, a few say things like “who is this?,” or “why are you sending me these emails?,” something which has Mr. Peck concerned his address has been used in a hack or attack of some sort. Most of those who have written something to him appear to be foreign, based on their names.

We’re trying to reach someone at Google about this, and will update with more as we have it.

Update: This article and headline were edited to clarify this is a glitch and related to Gmail, but until we can confirm with Google we won’t assume it’s the same glitch that caused the Gmail outage or is related to that outage. A variation of this bug was spotted earlier, according to Search Engine Land.

Since publication, we’ve received tips that a number of other users may be affected in similar ways as Mr. Peck – that is, it’s their address that is appearing when the link is clicked by Google searchers.

UPDATE, 6 PM ET: A Google spokesperson provided the following:

“Due to a technical glitch, some email addresses on public webpages appeared too prominently in search results. We’ve fixed the issue and are sorry for any inconvenience caused.”

Google has also confirmed that today’s glitch was not related to today’s Gmail outage.

—-

More Top Stories From TechCrunch

Gmail And Google+ Go Down Across The World, Service Returns After Roughly 50 Minutes

Talk About Timing: Google’s Reliability Team Sat Down For An AMA Right Before Gmail Exploded

Facebook Hilariously Debunks Princeton Study Saying It Will Lose 80% Of Users

Snowden Answers Our Burning Data Collection Question: What’s The Worst That Could Happen?

Apple Said To Be Working On Two Larger iPhones

Google Awarded Patent For Free Rides To Advertisers’ Locations

Where do you find the time for side projects?

Ever since I published the revenue numbers for my side-projects, I’ve been getting this question a lot:

When I answer, I’m tempted to make up something that sounds really impressive:

But I don’t do that. I also don’t currently use a really complicated time management philosophy. There’s a few things that I’m doing right now that have been helpful for me. If you’re like me (a parent and/or someone who has a full-time job) they might be helpful for you too:

1. Where are you going?

One of the best things I did this past year was making a decision to launch my book ”by the end of the summer”.

Setting a goal is so helpful; once you have a destination it really clarifies how you should be spending your time.

Your first step is to define what you want to achieve. I personally like 3 month projects – they’re smaller in scope, and easier to get going.

2. Jump aboard the inspiration train

If I get inspired, I try to start working on that idea right away. If I wait longer than a day, I lose the momentum.

This is especially helpful for cranking out an initial draft of your project. When you’re inspired, you have a lot of energy, you’re mentally alert, and you’re motivated. The first draft of many of my blog posts are written in fits of inspiration. I’ll blurt it all out as fast I can. Then, I’ll try to sit on it overnight, and come back and tweak it the next day.

3. What do I want to accomplish this week?

I don’t always keep this habit, but when I do, it’s really helpful. I use a Kanban board to write out a list of achievable tasks for the week. I put all those items in the “Backlog” column. Then, each day I pull up the list, choose one task (by moving it to the “Current” column), and work on it until it’s done.

Bonus: I’ve also found it helpful to have a shared Kanban board with other side-hustlers and JFDIers. I do this in Sprint.ly, but you can use Trello this way as well.

4. Ok, but where do you actually find the time?

I mentioned setting goals, using inspiration, and Kanban first because setting a good foundation is important. I’ve found having this foundation is more helpful than the actual logistics (which actually aren’t that exciting).

The short answer is: I work on side-projects whenever I have a spare moment. I sacrifice other things (watching TV, reading the newspaper, playing video games) so that I can do creative work.

I get most of my work done in the evenings, right after my kids go to bed, or early in the morning. My regular routine is to wake up early on Saturday and Sunday, and get 2-3 hours of work done before anyone wakes up.

I’ve also found it helpful to take my lunch hour (during the work week), head to a cafe and give myself 50 minutes to just write.

During these times, I eliminate distractions (Twitter, open tabs, notifications) and I focus on achieving just one thing.

Cheers,
Justin Jackson
@mijustin

Why Dogecoin is Important

What’s most interesting about Dogecoin is that it created a new way for cryptocurrencies to compete: on branding, rather than on technical merit. The dozens of preexisting Bitcoin alternatives had all changed a number here, an algorithm there. Their creators proclaimed them to be more secure, user-friendly, or economically fair than Bitcoin. Instead, the main contribution of Dogecoin’s creators was to name the coin after a silly internet meme and plaster pictures of shiba inu all over the place.

Frankly, it’s brilliant. It’s not surprising that there was a marketer involved in the coin’s creation, rather than a team of all software developers. And Dogecoin has shot up the ranks — at the time of writing, it has the 7th highest market cap out of the 80 cryptocurrencies listed at coinmarketcap.com, many of which have been around for much longer. In daily exchange volume, it has recently reached as high as second place.

Among all the possible brands to build a coin on, the doge meme has several advantages. First among them is that the currency can pretend to be a joke. As one friend put it: “Dogecoin pretends to be a joke currency pretending to be serious, but it’s actually a serious currency pretending to be a joke.” This lets its fans support it enthusiastically without looking unintentionally foolish; they merely look intentionally foolish. Doge is also fun and accessible, and there’s no threat to the brand from, for example, being associated with an online drug marketplace. If anything, that would make it funnier.

What other kinds of brands could work? Two imitators, Catcoin and CoinyeWest, have already come and gone. They aren’t different enough from Dogecoin to gain traction. But they do point in two interesting directions. For the first one, cats are a more generic and broadly familiar brand than the doge meme. Perhaps more people could get behind it. In that line, you can imagine coins based on national pride (USACoin!), ideology, sports teams, etc. And what CoinyeWest points us toward is coins backed by private, commercial brands. Pizza Hut could release a pizza-backed currency — far tastier than the gold standard. A DisneyCoin would be in trouble when the Silk Road decided to start accepting it. It’s not immediately clear that companies would ever prefer a decentralized virtual currency to a centralized one, but it’s worth thinking about.

The long-term success of Dogecoin is beside the point. It doesn’t have anything close to Bitcoin’s developer backing. And basing a currency on an internet meme presents its own risks: for example, the joke might get old. Those are two of many reasons that Bitcoin will keep its lead for now. But it will be interesting to see where and how smarter branding gets incorporated into new efforts.

Mondrian

Mondrian is a free vector graphics web app like Adobe Illustrator or Inkscape, available atmondrian.io.
Today I'm open-sourcing it.

Why?

I've spent the last year building Mondrian in private. It's at a pretty solid level of usability. It has been the most fun and interesting thing I have ever worked on, but I'm getting frustrated with the slow rate of progress I can muster by myself on weekends.

Come Help Build It

Artur Sapek
me@artur.co

New York City
January 24, 2014

I love San Francisco. I moved here from the UK about 4 years ago to start my last company, DailyBooth. It’s a wonderful city filled with interesting people doing amazing things.  It’s the city I grew up lusting over and when I moved here I wasn’t disappointed.

If you’re doing a startup and planning on raising money there’s probably no better place in the world to do it. However….

It’s no secret that rent in San Francisco is starting to get a little crazy. After living here for a while you start to lose context. Let me give you some.

The average price of a 1 bedroom apartment in San Francisco is now $2,800 a month. This is for a pretty standard, unexciting, ~700 SQFT space. That’s $33,600 per year. The average yearly income in 2012 was $44,321. I’ve been here for about 4 years. 4 * $33,600 = $134,400.

Enter St Louis.

The average rental price for a 1 bedroom apartment in St Louis is $720. 3.8x cheaper than San Francisco. Moving to St Louis is going to almost quadruple my company’s runway. You can buy a really nice 1 bedroom apartment in downtown St Louis for less than renting in San Francisco for 4 years.

Cheap cities are startup friendly

The leading cause of startup death is running out of money. Moving to a cheap city and doubling (or more!) your company’s runway will more than likely vastly increase your chances of eventual success.

There’s a tech scene here!

Sure, it’s not quite on the same level as San Francisco but it’s something.

• There’s this thing called Arch Grants which aims to attract entrepreneurs by giving them $50,000 in equity free funding. Marshall‘s last startup won one of these before being acquired. Applications are currently open for the next 20 grants!
• There are over 75 startups in our building alone.

Something to think about

We’re an internet company. We don’t need to be tied to a specific location. Hiring remote and having a distributed work force is far cheaper than hiring locally and making everyone come into an office.

If you’re based in San Francisco (or any other big, expensive city) and thinking of bootstrapping a company there are other options. I invite you to seriously consider St Louis. You’re even welcome to come and work out of our ($275 per month) office for a few days and we’ll show you around.

You’re not giving up living in a cool city

St. Louis isn’t just ridiculously affordable, there’s also a lot to keep you entertained.

St. Louis is neighborhood-y. We’ll be living and working downtown (pictured), but St. Louis boasts 78 other neighborhoods, each with their own flair.

Big money is going into the city. All the time. In only the past few years, billions of dollars have gone into renewing some of the City’s most recognizable institutions. The Peabody Opera House, Downtown Central Library, the St. Louis Art Museum  — up next, even the Arch itself — have all gotten updates and restorations in the past couple of years alone.

St. Louis repurposes things well. The exquisite restaurant and cocktail house Olio came out of an old gas station. You can take in your new movie releases on a couch in the Moolah Theater, an old Masonic Temple that also has a bowling alley in its basement . World-class climbing can be found at Climb So iLL, which used to be a hospital power plant building.

St. Louis has a great culinary scene. After a flourishing of microbreweries like Urban Chestnut, when the city’s original beer entrepreneurs got bought out, food trucks caught on, and then coffee roasters started a scene (including one of @jack’s favorite spots, Sump Coffee).

Most attractions are free. From spring to fall, weekends are chock-full of free festivals. Plus, its top-ranked Zoo, Science Center, History Museum, and all of the art museums are free to the public. The attractions that aren’t free — like the world-class Missouri Botanical Garden, the seriously unbeatable City Museum, or even the surprisingly relevant World Chess Hall of Fame — are more than worth the cover charge. And you never have to wait in line.

 Special thanks to Marshall‘s girlfriend, Tara Pham for the St. Louis tips.

If you found this post interesting you can follow me on twitter @jon and Need/Want @NeedWantInc.

Here in the eastern part of the United States, we’re huddling in subzero temperatures and dealing with the aftermath of a snowstorm. Still, there’s nothing to brighten our moods like the pristine beauty of a freshly fallen release–no, make that TWO releases. jQuery 1.11.0 and 2.1.0 are ready to keep you warm on these dark winter nights.

You can get the files off our CDN as always, and either use them directly or download them to your own server. Our download page has all the files and information you need, including pointers to the other CDNs that carry the files. Just give those folks a few days to update!

For those of you not following along for a while, both the 1.x and 2.x branches of jQuery support all recent modern browsers and have the same API. The 1.x branch, this time 1.11.0, adds support for the older versions of Internet Explorer (IE6, 7, and 8). The 2.x branch, today played by 2.1.0, adds support for non-traditional web environments like node.js and browser plugins for Chrome and Firefox.

jQuery went through a major house-cleaning with the 1.9 release that removed some features. If you haven’t yet moved from an earlier version, see the jQuery 1.9 Upgrade Guide and let the jQuery Migrate plugin do all the heavy lifting for you.

What’s New?

You may be wondering what great new things await you in these releases. Perhaps you’re fearing that they hold a bunch of breaking changes. You just know the project lead will suggest updating things right now. There goes your whole damn week and that trip to Florida. Well fear not! We’ve fixed quite a few bugs, but the other features and changes are mainly organizational ones that don’t affect the behavior of APIs. Your code shouldn’t break, it should just run a little faster. Here are the highlights:

Fewer forced layouts: In this release we declared war on places where we might inadvertently force the browser to do a time-consuming layout. We found a few and eliminated one in particular that could occur when changing class names. This can result in a big performance boost for some pages.

Granular custom builds: Our modularity is now defined by AMD, and it is easier to build small subsets of the library when space is at a premium. If you want to know more, we’ve hidden the details in the README file where nobody ever looks.

Lower startup overhead: The new modularity and avoidance of forced layouts led us to refactor our feature detects so that they run the first time they’re needed. If you never call the API needing that feature detect, you never run that code. Previously we ran all feature detects when the page loaded, that led to delays that were generally small, but added up–especially on mobile platforms.

Published on npm: Our releases will now be published on npm so that you can use them with node or browserify. Both the 1.x and 2.x branches are available on npm, but remember that only the 2.x branch is supported to run in node.

Published on Bower: We’re now using Bower for our internal dependency management including Sizzle, so you’ll see jQuery releases on Bower as soon as they’re available.

Some people have asked about supporting other package managers inside the jQuery library, but we’ve decided to only support the two that we use internally at the moment. There are more than a dozen package/dependency managers, it would be handy if they all could agree on a single format for projects to publish information. We don’t want the package manager’s overhead to be pushed off to individual projects like jQuery.

Although the glamor always seems to be in the new stuff, we don’t like to ignore the bugs and inconveniences that people have run across while using the last version. We worked hard to knock down our bug list and tackled quite a few of them. We even fixed a bug that only occurs in IE6, better late than never!

Sourcemap changes

This release does not contain the sourcemap comment in the minified file. Sourcemaps have proven to be a very problematic and puzzling thing to developers, spawning hundreds of confused developers on forums like StackOverflow and causing some to think jQuery itself was broken.

We’ll still be generating and distributing sourcemaps, but you will need to add the appropriate sourcemap comment at the end of the minified file if the browser does not support manually associating map files (currently, none do). If you generate your own jQuery file using the custom build process, the sourcemap comment will be present in the minified file and the map is generated; you can either leave it in and use sourcemaps or edit it out and ignore the map file entirely.

We hope to bring back and improve sourcemap support in the future, but at the moment neither the design nor the implementation seem suited for situations like jQuery’s, where there are widely distributed files on CDNs. We’d like sourcemaps (and browsers supporting them) to gracefully handle situations like file renaming or missing files. See our bug ticket for more information.

Acknowledgements

This release would not have happened without the hard work of many people. Thanks to everyone who reported bugs, tried out the prerelease files, or provided constructive criticism. Particular thanks are due to Alex Robbin, Amey Sakhadeo, Anthony Ryan, Aurelio DeRosa, Chris Antaki, Chris Price, Christopher Jones, Corey Frang, Daniel Herman, Domenic Denicola, Dominik D. Geyer, Forbes Lindesay, George Kats, Guy Bedford, Ilya Kantor, Jakob Stoeck, Jeremy Dunck, John Paul, Julian Aubourg, Jörn Zaefferer, Lihan Li, Marian Sollmann, Markus Staab, Marlon Landaverde, Michał Gołębiowski, Mike Sidorov, Oleg Gaidarenko, Richard Gibson, Rick Waldron, Ronny Springer, Scott González, Sindre Sorhus, T.J. Crowder, Terry Jones, Timmy Willison, and Timo Tijhof. Colin Snover’s commentary in #jquery-dev is also a source of rare humor for the team.

Changelog

jQuery 1.11 and 2.1 (common to both)

Ajax

Attributes

Build

Core

Css

Data

Effects

Event

Manipulation

Misc

Selector

Support

jQuery 1.11

Ajax

Attributes

Build

Core

Effects

Support

jQuery 2.1

Ajax

Build

Core

Event

Artist's impression VICTOR HABBICK VISIONS/SPL/Getty

The defining characteristic of a black hole may have to give, if the two pillars of modern physics — general relativity and quantum theory — are both correct.

Most physicists foolhardy enough to write a paper claiming that “there are no black holes” — at least not in the sense we usually imagine — would probably be dismissed as cranks. But when the call to redefine these cosmic crunchers comes from Stephen Hawking, it’s worth taking notice. In a paper posted online, the physicist, based at the University of Cambridge, UK, and one of the creators of modern black-hole theory, does away with the notion of an event horizon, the invisible boundary thought to shroud every black hole, beyond which nothing, not even light, can escape.

Peter van den Berg/Photoshot

“There is no escape from a black hole in classical theory, but quantum theory enables energy and information to escape.”

In its stead, Hawking’s radical proposal is a much more benign “apparent horizon”, which only temporarily holds matter and energy prisoner before eventually releasing them, albeit in a more garbled form.

“There is no escape from a black hole in classical theory,” Hawking told Nature. Quantum theory, however, “enables energy and information to escape from a black hole”. A full explanation of the process, the physicist admits, would require a theory that successfully merges gravity with the other fundamental forces of nature. But that is a goal that has eluded physicists for nearly a century. “The correct treatment,” Hawking says, “remains a mystery.”

Hawking posted his paper on the arXiv preprint server on 22 January1. He titled it, whimsically, 'Information preservation and weather forecasting for black holes', and it has yet to pass peer review. The paper was based on a talk he gave via Skype at a meeting at the Kavli Institute for Theoretical Physics in Santa Barbara, California, in August 2013 (watch video of the talk).

Fire fighting

Hawking's new work is an attempt to solve what is known as the black-hole firewall paradox, which has been vexing physicists for almost two years, after it was discovered by theoretical physicist Joseph Polchinski of the Kavli Institute and his colleagues (see 'Astrophysics: Fire in the hole!').

In a thought experiment, the researchers asked what would happen to an astronaut unlucky enough to fall into a black hole. Event horizons are mathematically simple consequences of Einstein's general theory of relativity that were first pointed out by the German astronomer Karl Schwarzschild in a letter he wrote to Einstein in late 1915, less than a month after the publication of the theory. In that picture, physicists had long assumed, the astronaut would happily pass through the event horizon, unaware of his or her impending doom, before gradually being pulled inwards — stretched out along the way, like spaghetti — and eventually crushed at the 'singularity', the black hole’s hypothetical infinitely dense core.

But on analysing the situation in detail, Polchinski’s team came to the startling realization that the laws of quantum mechanics, which govern particles on small scales, change the situation completely. Quantum theory, they said, dictates that the event horizon must actually be transformed into a highly energetic region, or 'firewall', that would burn the astronaut to a crisp.

This was alarming because, although the firewall obeyed quantum rules, it flouted Einstein’s general theory of relativity. According to that theory, someone in free fall should perceive the laws of physics as being identical everywhere in the Universe — whether they are falling into a black hole or floating in empty intergalactic space. As far as Einstein is concerned, the event horizon should be an unremarkable place.

Beyond the horizon

Now Hawking proposes a third, tantalizingly simple, option. Quantum mechanics and general relativity remain intact, but black holes simply do not have an event horizon to catch fire. The key to his claim is that quantum effects around the black hole cause space-time to fluctuate too wildly for a sharp boundary surface to exist.

In place of the event horizon, Hawking invokes an “apparent horizon”, a surface along which light rays attempting to rush away from the black hole’s core will be suspended. In general relativity, for an unchanging black hole, these two horizons are identical, because light trying to escape from inside a black hole can reach only as far as the event horizon and will be held there, as though stuck on a treadmill. However, the two horizons can, in principle, be distinguished. If more matter gets swallowed by the black hole, its event horizon will swell and grow larger than the apparent horizon.

Conversely, in the 1970s, Hawking also showed that black holes can slowly shrink, spewing out 'Hawking radiation'. In that case, the event horizon would, in theory, become smaller than the apparent horizon. Hawking’s new suggestion is that the apparent horizon is the real boundary. “The absence of event horizons means that there are no black holes — in the sense of regimes from which light can't escape to infinity,” Hawking writes.

“The picture Hawking gives sounds reasonable,” says Don Page, a physicist and expert on black holes at the University of Alberta in Edmonton, Canada, who collaborated with Hawking in the 1970s. “You could say that it is radical to propose there’s no event horizon. But these are highly quantum conditions, and there’s ambiguity about what space-time even is, let alone whether there is a definite region that can be marked as an event horizon.”

Although Page accepts Hawking’s proposal that a black hole could exist without an event horizon, he questions whether that alone is enough to get past the firewall paradox. The presence of even an ephemeral apparent horizon, he cautions, could well cause the same problems as does an event horizon.

Unlike the event horizon, the apparent horizon can eventually dissolve. Page notes that Hawking is opening the door to a scenario so extreme “that anything in principle can get out of a black hole”. Although Hawking does not specify in his paper exactly how an apparent horizon would disappear, Page speculates that when it has shrunk to a certain size, at which the effects of both quantum mechanics and gravity combine, it is plausible that it could vanish. At that point, whatever was once trapped within the black hole would be released (although not in good shape).

If Hawking is correct, there could even be no singularity at the core of the black hole. Instead, matter would be only temporarily held behind the apparent horizon, which would gradually move inward owing to the pull of the black hole, but would never quite crunch down to the centre. Information about this matter would not destroyed, but would be highly scrambled so that, as it is released through Hawking radiation, it would be in a vastly different form, making it almost impossible to work out what the swallowed objects once were.

“It would be worse than trying to reconstruct a book that you burned from its ashes,” says Page. In his paper, Hawking compares it to trying to forecast the weather ahead of time: in theory it is possible, but in practice it is too difficult to do with much accuracy.

Polchinski, however, is sceptical that black holes without an event horizon could exist in nature. The kind of violent fluctuations needed to erase it are too rare in the Universe, he says. “In Einstein’s gravity, the black-hole horizon is not so different from any other part of space,” says Polchinski. “We never see space-time fluctuate in our own neighbourhood: it is just too rare on large scales.”

Raphael Bousso, a theoretical physicist at the University of California, Berkeley, and a former student of Hawking's, says that this latest contribution highlights how “abhorrent” physicists find the potential existence of firewalls. However, he is also cautious about Hawking’s solution. “The idea that there are no points from which you cannot escape a black hole is in some ways an even more radical and problematic suggestion than the existence of firewalls,” he says. "But the fact that we’re still discussing such questions 40 years after Hawking’s first papers on black holes and information is testament to their enormous significance."

Forget about bitcoin. The latest go-to cryptocurrency is called dogecoin, a digital denomination that began life less than two months ago as a jokey tweet made by 26-year-old Australian Jackson Palmer.

But his joke has now taken on a life of its own. The total value of the market for dogecoin (pronounced dough-je coin) has just topped $US60 million ($68 million) and it has spawned a community comprising thousands of buyers, sellers, merchants, beggars, speculators and "miners", the people who mint the money.

Virtual economics: Jackson Palmer's jokey tweet took on a life of its own. Photo: Tony Walters

The current value of the bitcoin market is $US10 billion.

Advertisement

This week, transactions worth a total of $US14 million were made, including one Chinese investor who bought $US5 million worth of the virtual currency. And on a daily basis dogecoin transactions are outstripping those in the more established bitcoin market, albeit for a smaller overall value.

Mr Palmer, who is a product manager based in the Sydney office of the software company Adobe, is both amazed and elated by the experience of introducing so many newcomers to virtual economics.

The dogecoin market has become a networked, self-regulating, peer-to-peer community. Photo: Tony Walters

"If the world's got a problem at the moment, it's that people don't understand how economies operate, how scarcity works. And, if by joining the dogecoin community, they gain a better appreciation, then that's great."

To cap off a frenetic January, the community has also chipped in chipped in 27 million dogecoins - or $US30,000 - to help send the Jamaican bobsled team to compete at the upcoming Sochi Winter Olympic Games. (Based on exchange rates at the time of writing, one dogecoin is equal to $US0.0018.)

The Carribean team, immortalised in the 1993 movie Cool Runnings, is competing at its first Winter Games since 2002, after failing to qualify in 2006 and 2010.

Viral content: An example of the doge meme.

"We're supporting the under doges," Mr Palmer quips.

While there are still question marks about the long-term viability and security of these alternative currencies, associate professor David Glance from the University of Western Australia's Centre for Software Practice, for one, believes they have a big future - especially if they gain the backing of the official banking system.

"Ultimately, cryptocurrencies are no more crazy than shares in terms of investments," he says. "We can pretend that there is some logic behind valuations of shares but, in reality, most of it is just whim."

Mr Palmer's co-conspirator is Billy Markus, a software developer at IBM, based in Portland, Oregon. They have never met in person. But, after coming across the joke website Mr Palmer cobbled together after the tweet, he suggested adapting a prototype virtual currency he had been toying with, which was based on the bitcoin DNA.

At this stage, you may think that here are two smart alecs who have just invented and launched a virtual currency, so surely they must have helped themselves to a vault full of coins before opening for business.

Yes, they wrote the algorithms that manage the currency and set a market capitalisation limit of 100 billion. 

But on December 8, 2013, when dogecoin launched, they effectively lost control. Now the dogecoin market has become a networked, self-regulating, peer-to-peer community. 

These computer-generated currencies are produced through an activity known as mining where computers are used to process and solve mathematical puzzles and equations.

As the currency pool grows and heads towards its capped limit, more computational power is required putting the activity outside the reach of amateurs using standard equipment. Eventually, this activity is dominated by miners using supercomputers or banks of networked computer known as server farms.

"That's true virality," Mr Palmer says. "[Dogecoin] grew a mind of its own."

But that was not an issue for the founders. Mr Palmer says he and Mr Markus never intended that dogecoin get too serious or appeal to the spivs and get-rich-quick types who have gravitated more towards the bitcoin micro-economy.

In part, this is due to the large following dogecoin enjoys among users of reddit, a popular website known for its diverse and socially active community.

This is undoubtedly because dogecoin takes both its name and mascot from one of last year's most viral internet memes - or running jokes.

The doge meme takes its name from a reference on an episode of a popular 1990s internet cartoon series called Homestar Runner. The term "doge" was revived, mashed-up with a quirky photo of a pet Shiba Inu dog and then sprinkled with a few ungrammatical phrases written in the much-maligned comic sans font. Last year, it exploded across the internet.

Mr Palmer, who was born and raised in Gosford on the NSW central coast, believes the doge theme has helped to make this cryptocurrency market appear less menacing, more accessible and a place in which he hopes philanthropists will outnumber the profiteers.

"What I'd really like to see is for dogecoin to become the default tipping currency of the internet."

A monkey. That’s the default name a part in the JavaScript Engine of Mozilla Firefox gets. Even the full engine has its own monkey name, called Spidermonkey. 2013 has been a transformative year for the monkeys. New species have been born and others have gone extinct. I’ll give a small but incomplete overview into the developments that happened.

Before 2013 JaegerMonkey had established itself as the leader of the pack (i.e. the superior engine in Spidermonkey) and was the default JIT compiler in the engine. It was successfully introduced in Firefox 4.0 on March 22nd, 2011. Its original purpose was to augment the first JIT Monkey, TraceMonkey. Two years later it had kicked TraceMonkey out of the door and was absolute ruler in monkey land. Along the ride it had totally changed. A lot of optimizations had been added, the most important being Type Inference. Though there were also drawbacks. JaegerMonkey wasn’t really designed to host all those optimizations and it was becoming harder and harder to add new flashy things and easier and easier to add faults. JaegerMonkey had always been a worthy monkey but was starting to cripple under age.

Improvement of Firefox on the octane benchmark

The year 2013 was only eight days old and together with the release of Firefox 18, a new bad boy was in town, IonMonkey. It had received education from the elder monkeys, as well as from its competitors and inherited the good ideas, while trying to avoid the old mistakes. IonMonkey became a textbook compiler with regular optimization passes only adjusted to work in a JIT environment. I would recommend reading the blogpost of the release for more information about it. Simultaneously, JaegerMonkey was downgraded to a startup JIT to warm up scripts before IonMonkey took over responsibility.

But that wasn’t the only big change. After the release of IonMonkey in Firefox 18 the year 2013 saw the release of Firefox 19, 20, all the way to number 26. Also Firefox 27, 28 and (partly) 29 were developed in 2013. All those releases brought their own set of performance improvements:

- Firefox 19 was the second release with IonMonkey enabled. Most work went into improving the new infrastructure of IonMonkey. Another notable improvement was updating Yarr (the engine that executes regular expressions imported from JSC) to the newest release.

- Firefox 20 saw range analysis, one of the optimization passes of IonMonkey, refactored. It was improved and augmented with symbolic range analysis. Also this was the first release containing JavaScript self-hosting infrastructure that allows standard, builtin functions to be implemented in JavaScript instead of C++. These functions get the same treatment as normal functions, including JIT compilation. This helps a lot with removing the overhead from calling between C++ and JavaScript and even allows builtin JS functions to be inlined in the caller.

- Firefox 21 is the first release where off-thread compilation for IonMonkey was enabled. This moves most of the compilation to a background thread, so that the main thread can happily continue executing JavaScript code.

- Firefox 22 saw a big refactoring of how we inline and made it possible to inline a subset of callees at a polymorphic callsite, instead of everything or nothing. A new monkey was also welcomed, called OdinMonkey. OdinMonkey acts as an Ahead of Time compiler optimization pass that reuses most of IonMonkey, kicking in for specific scripts that have been declared to conform to the asm.js subset of JavaScript. OdinMonkey showed immediate progress on the Epic Citadel demo. More recently, Google added an asm.js workload to Octane 2.0 where OdinMonkey provides a nice boost.

- Firefox 23 brought another first. The first compiler without a monkey name was released: the Baseline Compiler. It was designed from scratch to take over the role of JaegerMonkey. It is the proper startup JIT JaegerMonkey was forced to be when IonMonkey was released. No recompilations or invalidations in the Baseline Compiler: only saving type information and make it easy for IonMonkey to kick in. With this release IonMonkey was also allowed to kick in 10 times earlier. At this point, Type Inference was now only needed for IonMonkey. Consequently, major parts of Type Inference were moved and integrated directly into IonMonkey improving memory usage.

- Firefox 24 added lazy bytecode generation. One of the first steps in JS execution is parsing the functions in a script and creating bytecode for them. (The whole engine consumes bytecodes instead of a raw JavaScript string.) With the use of big libraries, a lot of functions aren’t used and therefore creating bytecode for all these functions adds unnecessary overhead. Lazy bytecode generation allow us to wait until the first execution before parsing a function and avoids parsing functions that are never executed.

- Firefox 25 to Firefox 28: No real big performance improvements that stand out. A lot of smaller changes under the hood have landed. Goal: polishing existing features or implementing small improvements. A lot of preparation work went into Exact Rooting. This is needed for more advanced garbage collection algorithms, like Generational GC. Also a lot of DOM improvements were added.

- Firefox 29. Just before 2014 Off-thread MIR Construction landed. Now the whole compilation process in IonMonkey can be run off the main thread. No delays during execution due to compiling if you have two or more processors anymore.

Improvement of Firefox on the Dromaeo benchmark

All these things resulted in improved JavaScript speed. Our score on Octane v1.0 has increased considerably compared to the start of the year. We now are again competitive on the benchmark. Towards the end of the year, Octane v2.0 was released and we took a small hit, but we were very efficient in finding the opportunities to improve our engine and our score on Octane v2.0 has almost surpassed our Octane v1.0 score. Another example on how the speed of Spidermonkey has increased a lot is the score on the Web Browser Grand Prix on Tom’s Hardware. In those reports, Chrome, Firefox, Internet Explorer and Opera are tested on multiple benchmarks, including Browsermark, Peacekeeper, Dromaeo and a dozen others. During 2013, Firefox was in a steady second place behind Chrome. Unexpectedly, the hard work brought us to the first place on the Web Browser Grand Prix of June 30th.  Firefox 22 was crowned above Chrome and Opera Next. More importantly than all these benchmarks are the reports we get about overall improved JavaScript performance, which is very encouraging.

A new year starts and improving performance is never finished. In 2014 we will try to improve the JavaScript engine even more. The usual fixes and adding of fast paths continues, but also the higher-level work continues. One of the biggest changes we will welcome this year is the landing of Generational GC. This should bring big benefits in reducing the long GC pauses and improving heap usage. This has been an enormous task, but we are close to landing. Other expected boosts include improving DOM access even more, possibly a lightweight way to do chunked compilation in the form of loop compilation, different optimization levels for scripts with different hotness, adding a new optimization pass called escape analysis … and possibly much more.

A happy new year from the JavaScript team!

22 Jan 2014

NB: These techniques are universal, but for syntax we chose Postgres. Thanks to the inimitable pgAdminIII for the Explain graphics.

So Useful, Yet So Slow

Count distinct is the bane of SQL analysts, so it was an obvious choice for our first blog post.

First things first: If you have a huge dataset and can tolerate some imprecision, a probabilistic counter like HyperLogLog can be your best bet. (We'll return to HyperLogLog in a future blog post.) But for a quick, precise answer, some simple subqueries can save you a lot of time.

Let's start with a simple query we run all the time: Which dashboards do most users visit?

selectdashboards.name,count(distincttime_on_site_logs.user_id)fromtime_on_site_logsjoindashboardsontime_on_site_logs.dashboard_id=dashboards.idgroupbynameorderbycountdesc

For starters, let's assume the handy indices on user_id and dashboard_id are in place, and there are lots more log lines than dashboards and users.

On just 10 million rows, this query takes 48 seconds. To understand why, let's consult our handy SQL explain:

It's slow because the database is iterating over all the logs and all the dashboards, then joining them, then sorting them, all before getting down to real work of grouping and aggregating.

Aggregate, Then Join

Anything after the group-and-aggregate is going to be a lot cheaper because the data size is much smaller. Since we don't need dashboards.name in the group-and-aggregate, we can have the database do the aggregation first, before the join:

selectdashboards.name,log_counts.ctfromdashboardsjoin(selectdashboard_id,count(distinctuser_id)asctfromtime_on_site_logsgroupbydashboard_id)aslog_countsonlog_counts.dashboard_id=dashboards.idorderbylog_counts.ctdesc

This query runs in 20 seconds, a 2.4X improvement! Once again, our trusty explain will show us why:

As promised, our group-and-aggregate comes before the join. And, as a bonus, we can take advantage of the index on the time_on_site_logs table.

First, Reduce The Data Set

We can do better. By doing the group-and-aggregate over the whole logs table, we made our database process a lot of data unnecessarily. Count distinct builds a hash set for each group — in this case, each dashboard_id — to keep track of which values have been seen in which buckets.

Instead of doing all that work, we can compute the distincts in advance, which only needs one hash set. Then we do a simple aggregation over all of them.

selectdashboards.name,log_counts.ctfromdashboardsjoin(selectdistinct_logs.dashboard_id,count(1)asctfrom(selectdistinctdashboard_id,user_idfromtime_on_site_logs)asdistinct_logsgroupbydistinct_logs.dashboard_id)aslog_countsonlog_counts.dashboard_id=dashboards.idorderbylog_counts.ctdesc

We've taken the inner count-distinct-and-group and broken it up into two pieces. The inner piece computes distinct (dashboard_id, user_id) pairs. The second piece runs a simple, speedy group-and-count over them. As always, the join is last.

And now for the big reveal: This sucker takes 0.7 seconds! That's a 28X increase over the previous query, and a 68X increase over the original query.

As always, data size and shape matters a lot. These examples benefit a lot from a relatively low cardinality. There are a small number of distinct (user_id, dashboard_id) pairs compared to the total amount of data. The more unique pairs there are — the more data rows are unique snowflakes that must be grouped and counted — the less free lunch there will be.

Next time count distinct is taking all day, try a few subqueries to lighten the load.

Who Are You Guys, Anyway?

We make Periscope, a tool that makes SQL data analysis really fast. We'll be using this space to share the algorithms and techniques we've baked into our product.

You can sign up on our homepage to be notified as we take on new customers.

No "contract" was ever signed, this was Lars/King justifying their actions to a small indie developer that might otherwise have turned down the request to copy our game.

Scamperghost isn't the most original game in the world.  It's obviously inspired by Pac-Man but we at least took it in an original direction by making it a mouse avoider with no walls.

King.com, however, showed no respect for other people's intellectual property when they made a direct, blatant clone of Scamperghost. Now they've trademarked "Candy" and are using their massive legal power against other small competing developers. A bit of a double-standard, eh?

Junkyard Sam
junkyardsam@gmail.com

PS. Thank you, Squarespace, for the bandwidth. Please consider them if you need a webhost.