Performance Measurement Training: Developing Future Victim Specialists for American Indian/Alaska Native Victims of Crime Program

TINA DIMACHKIEH: Thank you so much for joining us for our Developing Future Victim Specialists Performance Measurement Training. I am Tina Dimachkieh, I will be your host for the day. I am a training and technical assistance specialist. I provide contractor support to OVC grantees across OVC programs in all things related to performance management and performance measurement of their OVC awards. 

Today's session, we will be going over why performance measure reporting is important to OVC, and what OVC does with the data that grantees are reporting in our systems. We then will be focusing on the Developing Future Victim Specialists reporting system, view the reports and the deadlines, and all reporting requirements that our grantees are responsible for. We, then, will shift over to focusing on the performance measures themselves and any updates for any grantees that have received DFVS funds in the past. And then, we'll conclude our session with a Q&A and providing contact information; so, you all know where to find us and how to reach out to us for assistance in the future. 

So, let's speak briefly about why performance measures and performance measure reporting is important to OVC. It's extremely important that our grantees focus on accuracy and timeliness of reporting on their performance measures and making sure that the data is as accurate as possible. This data is looked at very closely, and we generate plenty of different resources and reports from the data that you all provide to us in our systems. 

Providing performance measure reporting accurately and on time will help OVC demonstrate the value and the specific benefits of the different programs that OVC is responsible for and be able to showcase that information to Congress, to federal and state government agencies, to federal partners, to the victim services field at large, and to the general public and any other stakeholders that are involved in specific programs. 

Reporting accurately and on time also helps our data analysts at the performance management team be able to generate annual reports on the different programs and demonstrate the output of grant funding. We also are able to emphasize the progress made towards achievement of OVC's strategic and program goals and be able to really reach any target audience of grantees, federal partners, and agencies, and the general public. 

Here are a few examples of what we have been able to do with the data that's been reported by OVC grantees. Our team, our data analysts, specifically, have been able to create program data reports, such as the ones that you see on the screen here. We've been able to develop reports based on demographics, based on victimization types, and crime types, such as the topical snapshots, based on different regions across the country. We've also been able to develop different data collection tools based on feedback from our grantees who have been using the measures and the system to collect data on OVC funding. 

The team that is really reviewing your data, I know a lot of individuals don't really know where the data goes once it makes it past their submission in their report, but here is the team. I am the training technical assistance specialist, as I mentioned, but my team consists of data analysts and our Helpdesk Support, which is Harif, and our subject matter expert that connects with the developers, Liz Wain, and we're all led by our task lead, Virginia Ward-Proctor. We're all here to support you, whether it's today during the session or in general, as you complete your reporting requirements for your OVC award. What we can help you with is really just anything related to performance measures and performance management. 

Please be sure to reach out to us. We would love to hear from you. We love hearing from all of our grantees. We schedule welcome sessions to performance management and reporting for any new staff members or any new grantees that have not received federal funding in the past. We also schedule sessions to speak through the different performance measures and any data-related items, or any questions that you have around how your organization is reporting or collecting data. 

You can submit questions or concerns or suggestions for technical assistance through the Helpdesk, our PMT Helpdesk, and we'll be sure to get back to you and schedule a session with you. Our sessions range from 30 minutes to an hour, sometimes even some follow-up sessions, and sometimes just a 10-minute question that is easily answered over the phone versus an email. We also have our resource webpage. It's on the OVC website. You'll need to navigate to the "Performance Measures" page, and it's under the Transforming Victim Services Discretionary Grantee Performance Measures. That's where you will be able to find any DFVS resources for your specific program. And, once the session is posted, it will be posted on that webpage, as well. Okay, so let's focus now on the DFVS reporting system that you all will be using, the reports, and the deadlines. 

Most of our grantees should be familiar with the JustGrants system. It's the grants management system that provides applicants and grantees with an end-to-end experience throughout key parts of the grant management lifecycle. For our specific performance reporting purposes, this is the only system that our DFVS grantees will be reporting data into during reporting periods. You also will be submitting the progress reports directly into JustGrants moving forward for your performance measure requirements. There's a lot of other tasks and items that get completed in JustGrants, but, for today's session, we're only focusing on the performance measures and completing reporting. 

The types of reports that you'll be responsible for are, as you see on the screen, we have the Quantitative Performance Measure Report, and we have the Semiannual Narrative Report. That first report, the Quantitative Performance Measure Report, focuses on standard performance measure data. Think of this as the numbers report, where you're providing the data that you've collected about OVC-funded program activities that were completed over a 6 month reporting period. The data is entered into JustGrants directly within the question sets under the performance report for that reporting period. 

The Semiannual Narrative Report, this is the report where you'll be responding to narrative questions. It is considered to be a qualitative report where you're sharing information with your grant manager about program goals, activities, and other factors that impact program delivery. These responses are open-ended. They're in essay form and can contain numbers based on the data that you've shared in the Quantitative Performance Measure Report. These reports are completed in a Word document that is provided by the OVC Grant Manager to the grantees, and these reports, once completed, are attached to the JustGrants performance report before submission. I will be sharing more information on that process with you shortly. 

So, reporting due dates, our DFVS grantees are responsible for semiannual reporting of OVC data, and the first period is from January to June 30. The submission period for that reporting period is July 1st to July 30. Our grantees have 30 days to report on their grant activity for that semiannual period, and the deadline for reporting on that first period is July 30. The second reporting period is from July to December 30, with submission period being open in the system from January 1st to the 30th, and the deadline being January 30. 

The data that will be due for those two different reporting periods is the Quantitative Data Report, which are the question sets directly entered into the JustGrants system, and the narrative questions that are attached in that Word document. 

So, how do you locate your progress report? In order to locate a progress report, you will need to navigate to the "Performance Management" section once you open up your funding award and the award information. Once you navigate to the "Performance Management" section, you can open any of the performance reports that are available by selecting on the report number and be able to enter the report and see the status of the report. The status of the report is also available on the right-hand column. As you see on the screen, here, there are some statuses such as "Completed," there's "In Review," and there's "New." Any of those "New" reports would be the ones that you're required to complete. 

There's also a "Delinquent" status, if a report has not been completed and is now past the due date for completion. You also are able to see whether a report is a regular semiannual report or a final semiannual report, and you're able to see the reporting period for which that report is required and when the due date of each report is. Once you select on a performance report, you will enter that specific report, and you'll be able to access the question sets. 

As you see on the screen here with the circled link, that is a question set, and in order to begin completing them and entering data into the system, you'll need to select that link inside the performance report in order to be directed to the next page. Once you select the link, this is what the page will look like. You will be directed to the performance measures in order to complete them. All the questions must be answered in order to be able to move forward to the next question set. We will be talking about the different question sets that our DFVS grantees are responsible for, but you see them at the top of the screenshot here. You will get an overview of the question set, and then you'll get the grant activity and the other question sets. Once you complete the questions on a page, you can select the "Continue" button in order to move on to the next set of questions. And again, if you don't respond to the questions on a specific page, you cannot move forward to the next set of questions. 

\There is an option to clear selection. If you've made an error in selecting one of the options, you can clear your selection and start over, as well. 

Within our question sets, we have embedded into the system different validations. Those validations will be guided by your responses. Based on the response that you provide, the system will either show additional questions or hide additional questions from being shown to you based on that response that you provided in a prior question. As you see on the screen here, for the training question set, it asks for feedback surveys that were distributed. If you were to select "No feedback surveys were distributed," the system will hide any additional questions, and you'll be able to move forward to the next question set. 

However, if you were to select "Yes" to feedback surveys having been distributed, the system will then show additional questions, as you see in this page here, where you will be required to answer an additional set of questions. That is the validation set. This is the functionality in the system that we call a Show and Hide function. If you have selected yes, and you know that you're supposed to answer additional questions, but the information is not available to you, then you'll need to troubleshoot why that is happening. It could be a system error, or it could be something on your end with the browser that you are using. You'd need to reach out to the helpdesk to receive additional support. 

Okay, so now let's move on to having completed a question set. You'll hit the "Complete" button, and you'll be directed back to that main page for that performance report, and this is where you will submit your performance report; but, because you have a requirement of attaching a narrative, you will need to do that before you submit your report. It's a very simple process. You will go down, under the question set that you had already located, there's a section for attachments. In that attachment section you'll select on "Attachments," and you'll be able to open up a window where you'll attach anything that you need to attach from your computer, being that narrative document, that Word document, and you'll select "Attach." It will be, then, attached to your performance report, and then, you can move forward and select the "Submit" button. You see on the screen, here, for this specific report, there are the question sets that are completed. There's an attachment, a Word document, and now you can select "Submit." 

Please note that the system will not stop you from submitting your report if you don't have an attachment. That's not a rule in the system. You just need to make sure you're attaching a document. If you don't do that, your grant manager will not approve your report. They will change or send it back to you to ensure that you're attaching that additional report to satisfy the reporting requirements for the narrative reporting piece. 

For any additional resources, we encourage you to explore the JustGrants resources. The JustGrants team is a different team than our Performance Management Team. They handle the functionality of the platform, and they've created multiple really helpful resources for you all, such as the resources on their training webpage in general, or a reporting guide on how to complete reporting, how to access and complete the question sets, how to troubleshoot any errors that you're experiencing when it comes to question sets, and a guide on how to upload the additional documents that are a requirement for your program, and submitting the report. 

I briefly showcased how to do all that information, or provided information on how to do all these steps. However, I highly encourage you to review these resources once you start reporting on your award this coming January. Okay, so moving on to the performance measures themselves. We've talked a lot about the process, how to get it done, but let's focus now on the performance measures. Our Developing Future Victim Specialists grantees are responsible for these specific question sets that are shown on the screen, here. 

You are responsible for answering questions regarding grant activity, training, collaborative partnerships, strategic planning partnerships, shared measures, applicant measures, and that final Semiannual Narrative Question Report. Grantees should only be focusing on entering data as it relates to OVC-funded activities. You should not be reporting on the organization's activities as a whole, but only on items that took place, or activity that took place, with OVC funding spent. We are not looking for data as it applies to your entire organization. 

We're only focusing on the OVC-funded work that has been completed, so, any activity completed from other funding sources should not be included in the data that's collected. Within the questions, you will see two different types of questions. There are Baseline Questions and Current Reporting Questions. Some of the questions sets include a baseline question, but not all of them. I will be highlighting that for you, but just so you know, it's the collaborative partnerships, the partnership shared measures, and the applicant measures. Those are the three measures that include a baseline question. Some of these question banks, or these three question banks that include the baseline question, they're asking to gather additional information about activity that occurred prior to the grant becoming operational, so prior to any funding being spent. 

Baseline data for the performance metric is established at the beginning of the data collection process, and it supports an assessment of the initiative. It really is to evaluate what things looked like before OVC funding and what things are looking like after OVC funding. Grantees should enter the same baseline data for each performance report. 

So, for these baseline questions, within the different question sets, your response across reporting periods should always remain the same. If you've answered, let's say, with the number four for a specific baseline question in the first reporting period, that number should always be carried over to the subsequent reporting periods. The system will not do that for you. This is something that grantees should ensure that they have recorded and enter into the system every single reporting period. Unfortunately, there is no functionality in the system to have responses carried over from reporting period to reporting period. You'll have to enter that number again, but it needs to be the exact number that you've entered in the first reporting period, because the baseline responses should not be changing, because they are referring to activity that took place before the funding became operational, and that number should not be changing. 

The other type of question is the current reporting questions. Those are the quantitative and qualitative information, or questions about grant activity that takes place during the reporting period that you have open in the performance report, in JustGrants. 

Let's move on to talking about the question sets themselves. 

The first question set is the grant activity question set. This is a very straightforward question set with only two questions. The first question asks if this is the last reporting period during which the award will have data to report. The response to this question should always be a "No," unless you are, in fact, in the last reporting period or if you're terminating your award early for any reason, whether you've spent all the allowed funds, or there's some change that took place within your award that you are terminating early. 

The second question asks if there was any grant activity during the reporting period. This question is very important. It's asking if there was any funding spent during the reporting period, any funding that was obligated, spent or drawn down to implement objectives proposed by the OVC approved grant application. If you answer "Yes," it means that there is data that should be shared with OVC, and if you answer "Yes," that means that your grant should be operational or there should always be grant activity for every reporting period. However, if you answer "No," this notifies the system that there should not be any data reported. because there was no grant activity and there was no OVC funding used during that reporting period. And when you do so, [Pause] you will not need to respond to the additional questions, because you will not have any data to report on, because again, you're only reporting on grant activity as it relates to OVC funding spent. 

So, if you were to answer "No" to Question two, you do not need to complete the rest of the question set. The second question set is the training question set. This question set is to report on data for any training opportunities that were conducted by the grantee organization for interns under this program. The different data that you will be responsible for includes the number of hours of internal and external training delivered or completed by interns, the number of interns who attended or completed training, and information on credentials for training completion, and if you've delivered any feedback surveys, and if so, if they were distributed and collected, what the outcomes of those surveys are. 

The next question set is the Collaborative Partnerships question set. Here, grantees should report information on partner organizations that are participating in the initiative as a result of grant funding. The wording here is extremely important. It needs to be as a result of grant funding. Grantees are expected to provide information on any new groups, organizations, or agencies participating during that specific reporting period, and the total number of groups, organizations, or agencies participating during the reporting period, over reporting periods. 

So, an example would be if, in the second reporting period, you had a new organization join the initiative as part of grant funding, but you already had two other organizations in the prior reporting period that are now working with your organization on the initiative. Your response to the new groups would be one new organization, and then total number of groups participating should be three, because you have two carrying over from the last reporting period and one new organization that was reported on in this specific reporting period. 

The next question set speaks to strategic planning. This is where grantees are to provide information on planning documents that were completed during the reporting period. On the screen, here, you'll see the different types of planning documents that are listed within the question bank. We have mission and vision statements, advisory board charter, community partnerships, MOUs, internal needs or strengths assessment, community needs and strength assessment, program logic models, action plans, evaluation plans, sustainability plans, data collection plans, and standard operating procedures. If there are any additional planning documents that are not listed in our options for this performance measure, you have the ability to select "Other" within this category, and then you will be prompted to provide an additional description as to what that other item is that doesn't fall under any of these listed options. 

The next question set is the Partnership question set, and this question set includes a baseline question. The baseline question asks for the number of formalized collaboration agreements developed prior to grant funding, and then the questions set itself, for the current reporting, asks for information on formalized collaboration agreements developed and letters of support received that were funded as part of the agreement. The difference between formalized agreements and letters of support, is that formalized agreements are considered agreements when they are signed by heads of organizations with authority to commit resources such as time, dollars, staff, and facilities. Letters of support, on the other hand, lend organizational support, but do not commit any resources. 

Grantees are required to provide data on any formalized agreements developed, any letters of support secured, and the level of involvement at these partners that they are reporting on. For the level of involvement, it is actually in a table format, and you will select the level of involvement based on the options provided. 

The last performance measure question set is the Applicant Measures. For the specific question sets, grantees are to report information on application process and outcomes for the developing future victim specialist internships during that specific reporting period. There is a baseline question included in this question set as well. And again, baseline questions refer to providing data on that specific question set in the prior reporting period when the funding was not operational or before the OVC funding program had begun. 

For the specific questions set, you are to provide information or data on the number of applications that were received by the organization or organizations and those received from qualified applicants, the number of applicant interviews conducted, and number of applicants who accepted or were hired onto the organization. You're also providing information on the number of entrance or exit interviews conducted and surveys conducted and outcomes as well. The number of interns who started or completed an internship during that reporting period, the number of interns who indicated that they do plan to pursue a career in victim services, and the numbers of victims served at the host site, and those that were served specifically by interns. Those are all the questions sets that you're providing quantitative data on. 

I do want to speak a little bit to the Validations List and the different functionalities that are embedded into the system. As mentioned previously, the system has a functionality that we call the "Show and Hide" function. That Show and Hide function operates based on these validation lists that we provide. You have access to them and you can review them, but basically, a validation is a rule that's been entered into the system on the development side that will help the system ask the right questions of the grantees. 

So, if a grantee answers a question with a specific response, and there's a validation for that response, the system will then know if it should hide any additional questions or add any additional questions based on that prior response. 

So, I'm going to give you two examples. The first one is in the grant activity question set. As I mentioned before, if that grant activity question, the grantee had responded saying there was no grant activity for that reporting period, that specific 6 month period, the system will then know that the subsequent questions are not required to be completed, and the system will allow the grantee to submit that report without completing the rest of the report, whereas if you were to respond yes to grant activity, you are then required to respond to every single question within the question sets in order to be able to submit your report. Another example would be under the Applicant Measures for Question 3, for qualified applicants. 

If a grantee responded to the prior question, which is Question 2, stating that there were no applications for that reporting period, then the system will know to hide that additional question, or to make it not mandatory, because if there were no applications for that specific reporting period, then Question 3 is invalid. Asking for how many qualified applicants there are, if there were no applications, that next response would be "0," just as the question before it. 

So really, the easiest way to understand these validations, I'm going to show you the rest of the list for different questions sets, such as Training, the best way to understand the validations is to think of them as a set of rules to better calculate the data. I like to think of them as 1+1=2. If we have the validation stating that one plus one should equal to two, and then in your response, in your data, you list that 1+1=3, the system will flag that. it will either show a specific question or hide a specific question, based on the response provided. 

Now that we are done with the question sets and the quantitative reporting requirements, let's move on to the Semiannual Narrative Reports. Again, these reports are to be completed in a Word document. This Word document will be provided to you all by your grant manager with a set of questions within the Word document that you will be responding to. 

The information that you'll share with OVC will be extremely helpful to your grant managers, so please be as transparent and open as possible in your responses to these narrative questions. This will help your grant managers better understand what is going on in your programs, but also better advocate for your program and for additional funding and be able to speak to OVC leadership about the program and the importance of this specific funding. 

Information that you'll be sharing with OVC will be around the status of activities completed within the grant and any barriers to reaching each goal and objective. You'll be speaking about an overview of the program. You'll be speaking to performance measure analysis completed based on the data that you provided. You'll share information about any publications as it relates to the program, any accomplishments and achievements, program changes, any other activities that you'd like to share around the program, the program impact in your community and your organization, and any attachments such as photos, news articles, and reference documents. I like to think of these Semiannual Narrative Reports as a way to just really share the incredible work that your organization is doing and, in a way, brag about your organization and the level of effort that you're putting into this initiative. 

So, definitely think about it from that perspective. Share as much as possible through grant managers. Really get an understanding of what is happening with this initiative at your organization and at the local level. And with that, we have come to the end of our content. I did want to make sure that I shared the contact information with you all. There are two different helpdesks to reach out to. As I mentioned previously, we are the Performance Management Team. 

We support the OVC PMT Helpdesk. We're responsible for performance management and performance measures of your OVC reporting, or your reporting requirements. Here's our contact information. But, there's also a different helpdesk and that helpdesk is the JustGrants Helpdesk. They are responsible for the reporting platform that you all will be working in when you complete reporting. They are there to help you navigate the different functionalities of the system, to troubleshoot any errors, to identify roles, to reset passwords and troubleshoot any account issues in the system itself, whereas we are responsible for the performance measures and the data collection side of the house. 

So, here's both contact information, so you know who to reach out to. If you do accidentally reach out to the wrong helpdesk, we will just direct you to our colleagues at the other helpdesk, so no worries about that at all. But with that, we have reached the end of our session. We will be moving on to our Q&A session, and we will be stopping recording at this time.