Database Skillshare

Databases! Apparently they are useful. In a quest to better understand this gap in my knowledge, Rahul, Yu, Joshua, and I sat down.

There are so many tools out there. What you invest in learning is important because you don’t know what will still be around, especially the web-based ones.

Relational Databases

SQL as a way of querying a relational database – pain to learn, as it’s a programming language. Things like ScraperWiki help make that easier.
Can deal with your data in much more subtle ways. Worth it for the expertise and/or if you need to slice and dice to find stories in data.
SQLite is basically a database file on your computer. Can speak SQL to it. It’s behind your mail, your interactive nametags, etc. But if you want to share the data, you have to mail the file to someone. Doesn’t merge changes – have to treat it like a file. But for a one-off-research thing, it’s great. 10s of thousands of rows. Great for Python scripts, there are some UI tools (Lita – Adobe Air app). Research one-off
ScraperWiki can mod a file into an SQL file.
MySQL and PostGreSQL allow for security measures. Web apps. Runs on a server – you have to send queries to it, it sends things back. There are shortcuts to making this happen.
Xampp installs PHP, PHPmyadmin, Apache.
PHPmyadmin lets you talk to MySQL in a nice web-based UI.
Every row has a number. Numbers are unique per table. This is how they reference each other. Never put information in two places (unless you really need to).

Some Tools
SQL To learn the language, Scraperwiki.com really helps with playing around.
CartoDB is great for mapping things.
Import.io sets you up for easy scraping. So if you wanted to scrape moma.org, you would teach it what you wanted, and then tweak what it came up with.
Open Refine as difficult to learn but gives a table of charts as a correlation that plots column to column – helps you see if there’s a correlation between variables. Can group things for you based on different spellings because Google.
NaviCat does data transfer from one database to another.

Non Relational Databases

Joins are a pain! Screw that! Document database. Javascript Object Notation (JSON as a nice form of XML)
Don’t know what the structure will be yet. No standard way to query it. Write code to query, there are libraries and examples to help. Need a quick way to store data and fetch it.
When pulling in information, it’s nice when it’s already structured (like when Civic pulls from the Globe, they’re including word count, author, date, etc – some scripts were written for it, but…
Some Tools
CouchDB defines what is present, then pull and push. Difficulty is in querying it – there is no standard for that.
MongoDB updates and syncs well.
Firebase lets you play with these structures easily.

Spreadsheets
Tableau does some pretty awesome things. If you just want a visualization of the data in a spreadsheet.
Statwing also does the statistical analysis of the data you input.

Useful links
kkovacs.eu
http://datatherapy.wordpress.com/2013/10/24/tools-for-data-scraping-and-…
howfuckedismydatabase.com

Projects from the Boston Aaron Swartz Hackathon

Written with Erhardt GraeffSJ Klein

Intro Talk Ethan led us out by talking about the breadth of Aaron’s work, and what it is to be an “effective citizen.”

https://youtube.com/watch?v=1Pn3mm3bK6U

(Second day’s talks are written up already on the Civic blog here.)

Projects We Worked On We are deeply appreciative of all of the hard work done at the event, and about the social bonds built in our time together.

Ableson Report TL; DR Distillation and restating the Report to the President on MIT’s role (or lack thereof) in the arrest and suicide of Aaron Swartz, such that more people can join in the conversations around the issues specific to MIT.

https://youtube.com/watch?v=cJxP3jOdO-I

Finished with: Prioritized and referenced questions for the FAQ doc, this prezi (which can still be expanded upon, but this is a start), and a super pretty interface to put it all into as we work.

http://prezi.com/embed/9r2agd3u8ule/?bgcolor=ffffff&lock_to_path=0&autoplay=0&autohide_ctrls=0&features=undefined&disabled_features=undefined

Emerson Working with Mozilla’s web API structures to to wrangle control of personal data across the web back into the hands of users.

https://youtube.com/watch?v=VFjmcoWufRk%3Flist%3DUUbqhCwZfmeKwMhaXyW4tUtw

Repeal the CFAA Repeal the CFAA: The CFAA is broken; no one but prosecutors like it. Building a constructive, normative replacement, and strategies for getting support at all levels: executive, policy, law, prosecution, activists, cyberwar. Cooperating with Aaron’s Law and EFF work; but also tackling. Most discussions of “CFAA reform” have been incremental, in a framework of discussing what changes to current law are possible and would help fix recent problems; as opposed to describing why CFAA is broken and what proportionate and moral laws in that space wold look like. We’re trying to describe what effective policy would look like, starting from scratch. Policy, Legal, Social, Tech/Security, and Prosecutorial norms which make sense. Project details and analysis

https://youtube.com/watch?v=tr1g9MmOXZM

Strong Box A platform for whistleblowers to transfer documents to newspapers directly. Initial code by Aaron Swartz.

https://youtube.com/watch?v=SAGoft_e-PY

This gentleman fixed a bug in how to delete files which a journalist may no longer find relevant.

Tor2Web Set out to work on tor2web, which makes it possible for internet users to view content from Tor hidden services. It’s online in a (mostly) functioning form at http://tor2web.org. Worked on by Aaron Swartz.

https://youtube.com/watch?v=wzs-TScDqk0

Finished with: a WORKING simple anonymous editable pastebin (no public code yet; on AWS w/ sponsorship). Hot damn!

What’s Next? Keep going, of course!

Recurring Calls We’ll have an ongoing call—once a week on Mondays for November, then monthly. These are to touch base about our work, to help solve issues and celebrate successes. If you’d like to be added to the calendar, invite, please leave a comment or email Willow

Open Atrium We’ll track our progress and objectives on this open source project management platform. It’s a way for us to remain ambiently aware of each other while still being accountable. If you’d like to hop on a project, or be aware of how projects are progressing, check out atrium.aaronswartzhackathon.org

IRC As always, you can join us in the OFTC IRC room #aaronswhack

VizThink Overview

I do these live drawings while people are speaking in order to demonstrate their ideas. Orginally mentored by James Carlson, I started doing visual thinking in earnest when someone turned left in front of me, causing a shattered radius. Since, it’s become my primary method of note taking, and a wonderful way to augment written notes.

Different Things to Show

Charts!At its most basic, visual thinking is a way to show workflow and charts. Rather than explaining in lengthy and complicating words, a drawing can often demonstrate relationships and interactions between components. Charts can be serious and examinging:

Value Based Chart

Or they can be silly and humorous:

Dubstep

Individual IdeasOften, when people are speaking or beginning to flesh out an idea, it makes sense to draw the individual components or speaking points as just that – individual parts. Using Adobe Ideas on my iPad, I do one layer per point. Often, these just end up as a collection of strangely-shaped references to ideas, which can then be arranged (remember, different layers!) to look nice nested within each other. This is, I suppose, a form of graphic design. 

 If there are enough components, or enough detail, it’s worth embedding into a prezi and defining a path for the viewer. More on that later. And http://prezi.com/d6ciafhoptrq/intertwinkles/De Conflating IdeasIt also helps to de-complicate what is a part of a workflow, what isn’t, and where confusion is coming in. This is the part of the conversation or explanation where we start gesticulating or arranging things on the table to demonstrate a point. For instance, I was frustrated that a bunch of digital disaster response groups wanted to list all the other projects that were going on for a certain topic (in this case, Hurricane Sandy). This is bulky because then there are many places to update if a project changes, completes, or dies. After spending 40+ lines of text in chat trying to explain what I meant, drawing this picture helped much more.

One reason the internet is amazing is because of the ability to point sections of a webpage at other webpages (RSS FTW). Not doing that was over complicating matters, but so what trying to differentiate via text rather than in a drawing. 

Building a Story

Now that you’re able to deconflate ideas as well as delineate them, it makes sense to move on to how components of a story interact with each other. Rather than moving from node to node, this method takes layers in Adobe Ideas and stacks them on top of one another. In this way, you can start to see how ideas flow into each other, and how they interact.

http://prezi.com/zi19tv4lwypr/the-psychology-of-difference-and-the-science-of-difference/

System Interaction

Life isn’t linear, and at times it is difficult to express all the moving parts while not losing the trees for the forest. While all models are incomplete (but they can still be useful), having a drawing can acknowledge the boundaries of a systems model while not dwelling on those limitations. 

http://prezi.com/0tky5lswrnbn/fema-field-innovation-team/

Similar to storytelling and individual nodes, we still see the individual components. However, in a System Interaction drawing, we see how those components play off of each other, and see where leverage points might be.

Tools

Stylus

I use an iPad and an electrostatic sock over a chopstick from this dude: 

http://www.etsy.com/search?q=shapedad&view_type=gallery&ship_to=US

Sometimes I pick up a stylus from a store.

Adobe Ideas

Is what I draw in. It provides many useful components while not being so overwhelming as to be ungainly, nor difficult for new users.

https://itunes.apple.com/us/app/adobe-ideas-vector-drawing/id364617858?mt=8

Graphics are vector, though the canvas isn’t quite infinite. It exports as PDF, which can be surprisingly versatile once you get ahold of them. And, it imports nicely into Prezi. It’s also available on Android, but a bit laggy.

Notability

For general note-taking, and text-heavy talks, Notability is also pretty great. It’s what I use for some IEEE Global Humanitarian Technology Conference notes, because it nests drawings in with text in with images all in the same document. It’s not useful for the Prezi-zoom interface, but it’s much better for in-depth blog entries.

https://itunes.apple.com/us/app/notability/id360593530?mt=8

It also does this great thing where it will record audio and sync your notes to it. Personally super useful, especially as you become reliant upon including drawings in some way.

Prezi

Prezi is what a lot of this work goes into. It lets me guide the way people move through notes – nested ideas, eastereggs, and all. That is probably a tutorial in and of itself, but here is something about how my brain works around it: http://blog.prezi.com/latest/2013/10/4/how-a-broken-arm-and-prezi-helped-me-save-new-york.html

Oversharing as Digital towards the detail of Analog

Did you know there’s a level of Dots Per Inch after which your eye simply cannot see any difference? Any added level of detail isn’t perceptible to you unless you select an area to zoom in on, changing the inches over which the dots are distributed.

This is what came to mind when listening to the Oversharing Forum at Media In Transition 8 conference at MIT’s Media Lab. The speakers covered EverydayCarry, the panopticon, and quantitative self. The point was brought up of how you must always assume you are being surveilled, and it only takes one person in a group to be recording for the entire group to be documented. The responsibility was bandied from the person recording to the person being recorded, to the person sharing, to the spaces themselves have default settings for recording (or not) (think theaters vs conferences). Zittrain brought up that the only way to NOT be recorded is to not do anything notable, and that is a long dark path of social blandness and fragility.

If each of us is a set of pixels in an image, or we produce the pixels which make up a digital self, at some point you get high resolution by sharing more, but it’s still in the abstraction of viewing the whole picture that people get a sense of who you are. Strangely, because we are each sharing things with metadata, we are also able to get abstraction divorced from the individual, and rather across the topic (EverydayCarry being a great example of this). These pixels, if we each are keeping our heads down for fear of how we are treated in the future, lead to one bland picture when you step back from the individual into the zoom setting of society.

http://prezi.com/embed/9lu7dlnyl6vg/?bgcolor=ffffff&lock_to_path=0&autoplay=0&autohide_ctrls=0&features=undefined&disabled_features=undefined

For me, this is contingent upon two things: one is a celebration of diversity, so that image is beautiful regardless of zoom. The issue here is how we handle past mistakes, how we grow as individuals, and what it is to act for the sake of an act rather than as performance. Caveats of course apply for healthy vs antisocial deviations from the norm. Secondly, we have made explicit how individuals in aggregate form the social. The individual’s broadcast (or notable lack) has always been what has formed society, but now the ability to see and track this makes the sharing and examination itself as equally apparent as the information being shared. We must begin to detail where that line exists and the expectations and responsibilities associated with it.

Civics Education and Activism

Hi, I’m Willow, and I’m new to the Center for Civic Media. I’m a research affiliate, which I’m still not clear on the meaning of[1], but I’m looking at how flat organizations scale and I love it here. I used to think it was decentralized groups, but then I read Charlie’s excellent blog post and knew better (Charlie, who has recently shaved his beard and it is freaking me out). Before I was here, I was in Seattle – another Geek Promised Land – working with hackerspaces and education. I still juggle digital disaster and humanitarian response work with a group called Geeks Without Bounds. This is my first blog entry on the Center’s roll, and is flavored with my own ranty-pants tendencies – if you’re looking for the live notes of the event itself, they are linked below or back a few entries on this blog.

A topic which has rightfully taken up a significant portion of my brain cycles since arriving to Boston is, “what does civic engagement even mean?” In this vein, I had the honor of attending Harvard Law School’s Civic Education Conference today, and the sanity of sitting with the Center for Civic Media’s Erhardt. My own #vizthink notes were the usual stick figures, from which I took breaks to transcribe with the slap-stick comedy of co-live-blogging – typing over each other, inserted comments, egregious misspellings. Erhardt absolutely did the lion’s share of the typed live-blogging, and you should check out his entries.

There were four panels, all with prestigious panelees, and all with heart-felt commentary. Everyone agreed that education is broken, including (or especially) civics education. A focal point of the conference was that civics education has gone by the wayside, and that this is a strong systemic indicator of a larger cultural illness, and a cause of future national woes. The discourse of sixties were glorified (but no one talked about the activism), an understanding of balancing testing metrics and classroom autonomy was expressed, and the children were empowered alongside the instructors. Everyone will learn so much! And through that education, everyone will have a chance to speak! Confetti was thrown, and a unicorn manifested by the stage. (But seriously, you can check out the live notes for panels 123, and 4. Finding the unicorn should be cake.)

Let’s take a few steps back for a moment in this handy time machine I have just found in my pocket. The first conversation I had around civic engagement in Boston was with the amazing Jo Guldi, who pointed out that things like townhall meetings are therapy, they are not participation. Participation is striking, is protesting, is making sure you are heard even when those at the top don’t want to hear it. It is the demanding to have an active role in the world which you inhabit. Therapy is talking about what is going on and how it affects you, coming to terms and acceptance to those structures. In therapy, there is no assumption that what you care about will come into play. In response to this conversation, she wrote a great brief history of participation on her blarg, focused on participatory mapping.

This story is strewn with the wreckage of technologies for participation past. … All of them, originally, made similar claims — to create a more informed citizenry, to free expertise from the constraints of disciplinary prejudice, to incorporate the poor and disenfranchised in the political process, and to thereby enliven society.
Remembering the long trajectory of this process is important to discerning the difference between the hopeless reiteration of bad methods past and radical tools for transforming society.

Back to the more recent past[2]. During the panels, there was much talk about the cynicism of youth around civic engagement. Much hand-wringing around “if they only knew how to interact, they would!” Well, maybe, but only for certain demographics. Most of the youth of the day (and I’d argue most of the population of all ages) feel disenfranchised because they ARE, in fact, disenfranchised by that system. External to the traditional systems, there is lots of engagement happening, on platforms that those currently in power don’t fully understand[3]. But their approach is not to empower youth to build their own systems, mentoring them with institutional and historical knowledge. Their approach is to only listen to anyone speaking the right language in the right place (while wearing the right clothes, etc etc) and continue charging forward saying “we would have listened if only they had spoken.” Maybe they just don’t know how to listen, maybe the Illuminati exists and it’s totally intentional.

There was chuckling about how boring civics classes can be. Panelists talked about the sorts of discourse they had in school during the 60s around the Vietnam War. They talked about how having the ability to talk about controversial subjects made them better citizens. They spoke about a current lack of support for instructors in schools who wanted to address hot topics. Those in my row (myself included) wondered if that discourse had actually happened in the classroom, but agreed that controversial subjects are far more engaging and worthwhile to address, including (especially?) in the classroom.

One group which was represented well on the final panel (as well as in my heart) is Facing History and Ourselves, a fantastic initiative around how to speak about difficult aspects of our past. This not only gives a more realistic understanding of context and progression (which then allows us to act more knowingly in the present), but also trains students and instructors in how to communicate about difficult topics. So there are, in fact, people taking actions in the field around difficult discourse.

People spoke on the fear of polarizing classrooms and communities around issues. That most controversies are “vanilla’d” if allowed in at all. That we must learn to disagree without being disagreeable. But how are we to do this without practice? How are we to be exposed to differing viewpoints if not in school, the main purpose of which is exposure to new ideas and to be socialized? The ability to act upon your beliefs, and the ability to have respectful discourse while knowing where boundaries are, is an objective both challenging and utterly worthwhile.

The question I asked of the panel of Justices[4] was about this ability to ACT upon the knowledge gained. Their anecdotes about changing laws (the funny story of Grendel’s liquor license being the closest to home) were highly dependent not only upon their knowlege of systems but upon their access to levers of power. Students at Harvard are maybe not the most indicative of the rest of the US population’s ability to gain traction around their concerns. The answers I got were along the lines of knowledge being empowering enough[5].

Education IS a great place to start to bridge that gap, but what I heard today is doing it wrongTM. What was discussed today (except for the fourth panel, which did a much better job of acknowledging and bolstering current efforts[6]) would simply reify the current, broken system[7]. There was no place for dissent nor even honest discourse in what was proposed. Lip service was paid, but no route to action was offered. If a truly informed and empowered generation came out of a magically formed school system which incorporated these views on civic education, those individuals would be promptly jailed the moment they attempted to put those skills into use.

In short: don’t talk to me about being cynical about your old, busted system when the very way you refer to civic education doesn’t even involve engagement. You belittle the new future we are building, and in doing so you have no part in it. The population the majority of the speakers at this event have the objective of creating is a well informed, but still passive, one. A system which makes no space for being tested and questioned will rightfully crumble from its own rigidity and frailty. So how about you come play with us, instead?

My favorite example of continuum from our current education system into engaged citizenry is with All Hands Active in Ann Arbor, Michigan. They set up a program with a local school, where they went to teach the kids every once in awhile. Ideally, as those children age, they start attending the hackerspace of their own accord, contributing back to the community which helped to support them. They learn how to build more advanced things. And because the hackerspace scene is tinged with politics (please, American spaces, be more so in this), discourse has a route to engagement for all ages. Also, they play with insects and neuroscience.

1. Apparently I just get to hang out and read all day. And write things. Who does that?! I am the luckiest.
2. 
3. Say, for instance, so many of the things the Center has to do with. Go team us!
4. Side note: don’t shout “CHECK YOUR MIC” at a bunch of Justices while being a blue-haired punk-patch wearing kid, nevermind the bowtie. It leads to the same sort of anxiety jumping out of your car at a military checkpoint to get your ID from the trunk causes.
5. Insert privilege rant here.
6. Including the Tribunal Ban system in League of Legends. Be still my geeky heart. This is a game where you are paired with 3 other random people, and if someone on the team treats the others ill, a case can be brought against them. Seriously fascinating, given the utter lack of “civics education” many of the participants have had in any formal system.
7. This is not to say I’m not glad the conversation is being started. But the Cancer Walk isn’t what cures the cancer.