Evaluation and Research
Background | Purpose of the evaluation | Approach to evaluation | Domains for evaluation | 1. Organisation and structure | 2. Systems and processes for providing integrated health care | 3. Services provided and outcomes of care | 4. Relationships to the broader health care system and other health and community services | Implementing the evaluation | Sources of data | Appendix 1: Suggested questions to address the 4 domains | Appendix 2: Objectives of the HealthOne NSW program and relationship to domains of the evaluationBackground
It is now generally accepted, both nationally and internationally, that health systems must place greater emphasis on reducing the demand for hospital services by strengthening primary health care. This requires effective strategies for health promotion (maintaining wellness) and disease prevention, as well as earlier diagnosis and intervention to reduce health risks and avoid or delay the development of chronic illnesses.
The historical Australian models for delivering primary health care cannot easily accommodate theses imperatives. This is particularly true for Australians in poorer socio-economic circumstances whose health outcomes are worse than those of other Australians.
The HealthOne NSW program represents a new approach to providing primary health and community care services in NSW. It is an important element of the NSW Health Integrated Primary and Community Health Policy 2007-2012 which sets the direction for the primary and community health sector in NSW for the next five years and beyond.
The HealthOne NSW program aims to develop a strong, integrated and sustainable primary health and community care sector which is capable of delivering patient-centred, comprehensive and coordinated care to local communities. It takes a continuum of care approach, with a deliberate focus on health promotion and protection to ease the burden of ill health, especially that of chronic disease, and the reliance on acute care services.
The model of care revolves around teams of health professionals delivering primary health care, with each member of the team using their specific skills to assist in the care of an individual person. The creation of the required teams will involve the integration of public sector services (generally salaried community health staff employed by Area Health Services), private sector general practice (subsidised through Medicare Benefits Schedule, largely under fee for service arrangements) and other health care professionals and services.
HealthOne NSW services are therefore expected to:
-
Support individuals and the community to make healthy choices, take greater responsibility for their health, and contribute to health service planning and development.
-
Deliver accessible, affordable primary health care services that are responsive to the health needs of the local community, systematically identify and manage individuals and groups at risk, and reduce inequities in health status.
-
Provide coordinated, continuing and integrated care with effective links between the primary, secondary and tertiary health care sectors.
-
Recognise the central importance of the broad determinants of health and develop cross-sectoral processes, structures and initiatives in collaboration with other agencies and providers.
-
Link service provision, evaluation, research and health workforce education and training to provide a sound basis for flexible and responsive service development and professional development.
-
Establish a professionally satisfying and financially sustainable working environment that encourages the recruitment and retention of a high quality, flexible primary health care workforce.
Purpose of the evaluation
HealthOne NSW represents a significant investment by the NSW Government ($40 million over 5 years) in shifting the orientation of the health system more towards providing treatment, care and support in the community.
The evaluation will determine the appropriateness of this policy direction and the effectiveness of the investment and will therefore guide future developments in primary health and community care in NSW and beyond.
The evaluation has two main purposes:
-
To support the continued development of the program over its current life, by providing information on:
-
how it is being implemented
-
what helps and hinders integration of services and of patient care
-
how problems have been solved in different contexts.
-
-
To inform decisions about the future of HealthOne NSW through:
-
determining the extent to which integration facilitates the aims of HealthOne NSW, in particular the provision of health promotion, disease prevention, early intervention and continuing, coordinated care in the community.
-
documenting successful strategies
-
exploring the enthusiasm of key stakeholders (health professionals, patients and the communities the services serve) for the model of care
-
determining the impact and outcomes of the program
-
Approach to evaluation
As HealthOne NSW services are being established de novo, they will pass through a series of developmental stages. The quality framework developed by Donabedian is therefore proposed for this evaluation, with its focus on structure, process, output and outcomes as follows:
The structure is the organisation of the service including the human and physical resources and arrangements for team care.
The process is the process of care and concerns the arrangements in place to support team care.
The outputs are who gets what services when and how they fit with the service objectives.
The outcomes are the impact on health status, which result from the integrated health care delivery arrangements.
Domains for evaluation
Drawing on Donabedian's framework, the following for domains should be addressed in the evaluation.
-
Organisation and structure
-
Systems and processes for providing integrated health care
-
Services provided and outcomes of care
-
Relationships to the broader health care system and other health and community services.
Each of these is outlined below. Appendix 1 suggests questions that might be addressed in developing and implementing the evaluation. Appendix 2 maps these areas of evaluation against the goals, aims and objectives of the HealthOne NSW program.
1. Organisation and structure
A core question for evaluation is whether the service has an organisation and structure that can support an effective, integrated and sustainable health service. To answer this, the evaluation will need to:
-
Describe the organisational and financial structures that are established;
-
Assess whether they are financially sustainable from the perspective of the different stakeholders, and on what assumptions; whether they can operate as effective parts of the state and Commonwealth funded health sectors;
-
Assess whether these structures can employ and support staff from a range of professional and organisational backgrounds;
-
Describe the clinical and business governance arrangements in place.
How well organisational structures support integrated care will determine the potential for ongoing development of services and commitment of stakeholders over time.
2. Systems and processes for providing integrated health care
The evaluation will need to develop an understanding of the range of staff and their roles, and the systems that are used to share information, coordinate care and ensure that care is high quality. It will also need to determine the extent to which each HealthOne NSW service operates internally as a multidisciplinary team and how it coordinates patient care with other external services. Both staff and patient perspectives are required about how care is planned and implemented across different team members, and how well the systems for communication and sharing information support teamwork and coordination of care.
3. Services provided and outcomes of care
The evaluation will review the extent to which HealthOne NSW services reflect the 'model of care' and meet the needs of patients and the local community. In particular:
-
Whether the care provided is comprehensive and patient centred
-
Whether care complies with accepted standards for 'quality' care
-
Whether it includes prevention and early intervention activities, as well as continuing care and management of chronic disease
-
The accessibility and affordability of the HealthOne NSW services
-
The impact on access to community health type services
-
The impact on health outcomes (to the extent that it is possible to measure this within the relatively short intervention period)
This part of the evaluation will provide an opportunity to examine the balance between the different elements of the model of health care that the HealthOne NSW services have adopted.
4. Relationships to the broader health care system and other health and community services
The HealthOne NSW services will operate as part of a broader health care system. Therefore it is important to identify how they link to the broader health system at local, regional, state and national level. Locally this will include:
-
Information sharing and referral pathway arrangements
-
Use of support services including those from Divisions of General Practice and Area Health Services
-
Links with other organisations such as councils, self help groups and community care services
-
Impact (both positive and negative) of HealthOne NSW services on other GP, community health and related services in the region
-
Links to training institutions, especially universities, which will provide access to research, teaching and training opportunities through participation in education programs and development of opportunities for student placements.
Implementing the evaluation
The evaluation of HealthOne NSW will occur concurrently with the development of the services. As new ways of working and new collaborations are developed, they are likely to evolve over time in ways that are not entirely predictable.
In this context, the evaluators will need to be able to address the challenges of evaluating HealthOne NSW services that will differ in:
-
Size
-
The extent of innovation in organisational development (for example, from complementing existing general practices with community health staff to creating a new health service organisation)
-
The integration model (for example virtual integration, co-location, hub and spoke)
-
The health and/or workforce priorities to be addressed, according to local need
-
Commencement dates
The way in which HealthOne NSW services function as part of the wider network of health services will be most relevant when the services are well established.
The evaluation should take an 'action research' approach to ensure that there is scope to inform the development of services over time. This will involve feedback to individual HealthOne NSW services, NSW Department of Health and other interested stakeholders. This feedback would need to be tailored to local interests and provided in a timely fashion to be incorporated into the next stage of development of the program.
The evaluation will need to be flexible enough to capture the variation between sites, but will also need to have a common core that allows for comparisons across sites and an overall summation of the effectiveness of the program. The common core will ideally be reflected in a minimum data set which services would be asked to collect, using electronic record systems wherever possible.
Finally it is important to recognise the heavy clinical demands on service providers and ensure that any evaluation activities that are the responsibility of services are done in such a way as to not overburden already busy clinicians. It will also be important for the evaluation to be seen to contribute to the development of the service.
Sources of data
The evaluation will depend on five main sources of data:
-
Review of relevant documents
-
Interviews with staff
-
Data from clinical information systems
-
Patient questionnaires and interviews
-
Interviews with other local stakeholders
The evaluators will work with the sites to organise data collection through identification of relevant documents, extraction of clinical data from practice electronic information systems, organisation of practice visits and interviews with HealthOne NSW staff and other stakeholders.
Interviews with staff and patients, and other stakeholders will provide information on aspects of the services that are additional to information extracted from practice data management systems and document review.
Data from clinical information systems will be critical as this may be the only source of patient level data. How far this extends will depend upon the sophistication of clinical information systems.
Getting the patient perspective is important. Initial contact with patients would need to be made through the clinical team to ensure appropriate consents are obtained.
Appendix 1: Suggested questions to address the 4 domains
- Organisation and structure
Core question: Does the HealthOne NSW service have an organisation and structure that can supportan effective, integrated and sustainable health service?
1.1 What organisational structures have HealthOne NSW partners set up?
1.2 Are these financially sustainable?
1.3 Are these able to operate as effective health service structures within the policies and rules of the NSW and Australian health care system?
1.4 Are these structures able to utilise staff from a variety of professional backgrounds or organisations and address their work related needs and professional support and development?
1.5 Do the centres have effective systems for clinical and business governance?
-
Systems and process for providing more integrated care
Core question: Does the HealthOne NSW service have systems and processes that enable staff to provide integrated care?
2.1 What roles do different staff members have in relation to providing or supporting patient care?
2.2 What systems and processes have the HealthOne NSW services established to support coordination of care, and how well do these support team work, care coordination and responsiveness to the needs of patients
2.3 What systems and processes have HealthOne NSW services established to support participation in training and research?
2.4 To what extent does staff work as a multi-disciplinary team both within the HealthOne NSW service and with health and other related organisations outside the service?
-
Services provided and outcomes of care
Core question: Do the services provided by the HealthOne NSW service meet the needs of patients and the local community?
3.1 To what extent do HealthOne NSW services provide appropriate care, in particular care that is:
3.2 To what extent does the HealthOne NSW service take part in population health / health promotion projects addressing the determinants of health?
3.3 Does the service cater to those most in need in the local area, and are there arrangements to ensure that they are affordable to all?
3.4 Does having community health staff providing services from a HealthOne NSW service make these services any more or less accessible and affordable to the local population?
3.5 What impact on health outcomes has the service had (at a local community and individual level)
-
Relationship to the broader health care system/ related health and community care services
Core question: How well does the HealthOne NSW service function as part of the wider network of health and health related services?
4.1 To what extent do HealthOne NSW services use and/or contribute to established information and referral systems
4.2 To what extent do HealthOne NSW services access or contribute to support services, including those from Divisions of General Practice and Area Health Services
4.3 To what extent do HealthOne NSW services take part in relevant local, regional, state and Commonwealth initiatives
4.4 How do HealthOne NSW services link with relevant services/organisations in other sectors (eg local councils, self help groups, community care services)?
4.5 To what extent do HealthOne NSW services take part in teaching/training or research activities?
4.6 What impact does locating community health staff in a HealthOne NSW service have on any other community health and related services in the area?
Appendix 2: Objectives of the HealthOne NSW program and relationship to domains of the evaluation
The table below maps the suggested evaluation against the aims and objectives of the HealthOne NSW program.
| Objectives of HealthOne NSW | Evaluation question |
|
Objective 1 - support individuals and the community to make healthy choices, take greater responsibility for their health, and contribute to health service planning and development. |
3.1 3.3 |
|
Objective 2 - deliver accessible, affordable primary health care services that are responsive to the health needs of the local community, systematically identify and manage individuals and groups at risk, and reduce inequities in health status. |
3.2 3.4 3.5 |
|
Objective 3 - provide coordinated, continuing and integrated care with effective links between the primary, secondary and tertiary health care sectors. |
2.2 2.4 3.1 3.3 4.1 4.4 4.6 |
|
Objective 4 - recognise the central importance of the broad determinants of health and develop crosssectoral processes, structures and initiatives in collaboration with other agencies and providers. |
3.2 3.3 4.4 |
|
Objective 5 - link service provision, evaluation, research, and health workforce education and training to provide a sound basis for planning, flexible and responsive service development and appropriate health professional education. |
2.3 4.5 |
|
Objective 6 - establish a professionally satisfying and financially sustainable working environment that encourages the recruitment, retention and ongoing professional development of a high quality, flexibleprimary health care workforce |
1.1 1.2 1.4 2.4 |
This web page is managed and authorised by Inter-Government & Funding Strategies of the NSW Department of Health. Last updated: 30 March, 2009
