All of the outcome indicators that you would like to track as a partnership may not be readily available. Part of the work is directly with partners to help develop shared measures and determine the best way to start tracking them. One example from StrivePartnership’s early work was in selecting our outcome indicator for goal one: Every child will be prepared for school.

There were a number of indicators that we could potentially track (infant mortality, low birth weight, pre-K experience, etc). But we knew the best measure would be one that is a close proxy to the goal, and the one we landed on was percent of children who are assessed as ready for school when they enter kindergarten. But this data wasn’t consistently available. We worked closely with the Success By 6® early childhood networks and the school districts to land on an assessment and begin tracking the data on a regular basis. The early childhood networks were meeting on a regular basis, and I remember that in one of the Covington network meetings, we brainstormed a list on flipchart paper of about 20 different assessments that were being used by partners. There are no common measures for kindergarten readiness and there aren’t even standard definitions of it. Many factors influence a child’s readiness for school including cognitive development, physical well-being, language use, approach to learning, motor development and social/emotional skills. But we needed to land on something as a population-level measure, even if the measures are imperfect, in order to advance the conversation. “Don’t let perfect be the enemy of good” was a mantra, and we had three good measures in our three geographic areas: Cincinnati, Covington and Newport.

Cincinnati is using a brief assessment tool called the Kindergarten Readiness Assessment – Literacy (KRA-L), which helps teachers identifies early reading skills. It is an assessment that has been adopted by the state of Ohio. Newport started out using the Developmental Indicators for Early Learning (DIAL-3) screen tool. The DIAL-3 provides scores for motor, concepts and language. Covington started out using the Dynamic Indicators of Basic Early Literacy (DIBELS), which measures letter-naming fluency and initial sound fluency.

The work of a data manager could involve getting into the weeds on all these measures with members of the collaborative action networks where necessary, helping to sort through the various measures.The partnership can highlight the discrepancies in the ways school readiness is measured and help advance the conversation around the importance of good data and in pushing toward common measures. Covington and Newport agreed on and began reporting a common measure (the Dial-3) a couple of years after the initial baseline report. And just recently, Kentucky adopted a new statewide assessment that all districts in the state will be using.

As mentioned earlier, there are three primary areas where I found myself spending time on any given day: digging into data, building relationships and consensus with stakeholders, and supporting the data needs of collaborative action networks.  A short list of competencies for a data manager would include:

• Knowledge and demonstrated success in data collection, management and analysis; knowledge of education and community data resources
• Ability to build relationships and work with key partners to develop a comprehensive community accountability system that incorporates data across the cradle-to-career education pipeline
• Ability to address and overcome uncertain and complex issues to achieve desired results
• Plans for the collection, analysis and reporting of data to measure the partnership’s impact and to facilitate evidence-based decision-making and continuous improvement
• Develops and cultivates relationships with community partners and stakeholders, including data and research professionals in education, business, faith, nonprofit, philanthropic and civic sectors

See A day in the life of a data manager: part one

The post A day in the life of a data manager: part two appeared first on StriveTogether.

As a data manager for a StriveTogether Cradle to Career Network partnership, there are three primary areas where I found myself spending time on any given day: digging into data, building relationships and consensus with stakeholders, and supporting the data needs of collaborative action networks.

Evidence-based decision-making is an underpinning of a collective impact partnership, and one of the first things to tackle after establishing the shared vision and goals is to establish a set of shared outcome indicators to help measure progress toward the shared goals. And so a data manager needs to dive head first into the data itself and really understand all the sources, variables and caveats to how the data may be collected and presented. We started with a list of over 75 potential measures, and so the data manager really needs to understand the data landscape in order to be able to help steer the data team and partnership in getting to consensus in narrowing that list down. The manager also needs to become the local education data “expert” and help build credibility for the partnership by being one of the go-to people for questions related to education data and results in the community.

Building relationships and consensus, however, is just as important as the data analytic skills. A partnership’s director and local champions will definitely help with building relationships among partners and advocating for data transparency, but the data manager also has to be able to forge relationships with the key data partners and build trust with them. One of our first efforts was to form a data committee comprised of all the data experts from key partners at the table: the school districts, postsecondary institutions, early childhood professionals and other community data experts. As a committee, we came to a list of 10 shared outcome indicators together, using a set of criteria that we developed, to take back to the executive committee as a recommendation.

It is important that this process is done with your key partners as opposed to it feeling like you are producing a report about your key partners. And so establishing relationships and building trust are key ingredients in this. Landing on the indicators is a back-and-forth process of presenting ideas and getting feedback until you have built something together that everyone feels ownership of. As a result, when we released the first report, executive committee members could speak with confidence about it, knowing that they had truly helped to create it.

Coming next: Working with networks to define indicators where no clear ones exist and key competencies of a data manager: See A day in the life of a data manager: part two.

The post A day in the life of a data manager: part one appeared first on StriveTogether.

The question posed around how to choose the right outcome indicators was a really good one that I’ll explore a little here. And it’s one that comes up a lot. How do you know that you have the right indicators? What if there is another critical measure that you missed at sixth grade, for example?  The reality is that there are a lot of important indicators that can be tracked and reported on. We started off with a list that was close to 100 potential indicators, but this is of course way too many. Not only is it too many to feasibly track on a regular basis, but the fewer indicators you have, the more focused you can become and the easier it will be to focus a wide collective impact effort to drive a core set of outcomes.

One of our mantras, espoused by the partners in creating the first StrivePartnership report card in 2008, was don’t let perfect be the enemy of good. This was a critical disposition, because for the outcome indicators that we settled on (we started with 10 in the first report), there were any number of reasons why a particular indicator didn’t really tell the whole story. As one example of many, the kindergarten readiness assessment data in Ohio only measures literacy. However, many will argue that we need to be measuring more than this to understand if children are prepared for school. So, we’ll work on advocating for a better measure (and the report card also becomes a good tool for this purpose), but we won’t let this stop us from publishing a good indicator that still gives a strong proxy to the ultimate goal that every child be prepared for school. We created a data committee to help develop the report and one of the very first things we did was to create a criteria for selecting indicators to help guide this process.

So how did we get from 100 indicators down to 10? We started with the Cradle to Career Roadmap to Success, which includes the benchmarks (the research-based competencies and experiences that act as predictors along the roadmap, indicating future success along the continuum). Then the key transition years are time periods along the continuum where students are often found to fall at risk if they are not developmentally on target. The combination of benchmarks and transition years led to the creation of the five big overarching goals (prepared, supported, succeeds, enrolls and graduates), and the indicators developed from these as measures of how well we, as a community, are achieving these goals.

Just because you don’t choose an indicator doesn’t mean that it’s not important, and it may be tracked in some of the strategic priority areas of the partnership. The outcome indicators that we are interested in are primarily the longer-term indicators that tell us if we’re getting to our goal. The short- and mid-term indicators are still important, but not the primary focus in choosing indicators for the report card.

And the development of the indicators was very much a collaborative process with all the partners. The data committee was comprised of the lead data people from each sector contributing data — early childhood, k-12, higher ed — as well as some key community partners, including United Way and Children’s Hospital. We had a series of many meetings in the first year in an iterative process to develop and refine our list of indicators. And we have refined (and hopefully improved) the indicators and reporting style in each subsequent year.

We have since reduced the 10 indicators down to eight. And the work of the entire StrivePartnership is focused on helping to move the dial on these core outcomes:

• Percent of children assessed as ready for school at kindergarten
• Fourth-grade reading achievement
• Eight-grade math achievement
• High school graduation
• ACT composite score
• Percent of students enrolling in college
• Postsecondary retention
• Postsecondary completion

Ultimately, choosing the right indicators is both a science and an art. A science in that you need to have a rigorous process and criteria for selecting indicators, and a good fundamental understanding of what data is available and what it can tell you. And an art in that it’s also about aligning this work under the shared vision and leadership of the overall effort, as well as building relationships with the data experts and owners in order to create the buy in and shared ownership to move the work forward together.

The post Developing shared measures appeared first on StriveTogether.