Mission report of Activity 4.5 Pilot evaluations

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 4. Standards and Guidelines for QA in HE

Activity: 4.5 Pilot Evaluations

Name of the Expert: Ms Hilla Aurén

Dates of the Mission: 20 – 24 February 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

      Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

International cooperation in quality assurance has been an essential element of the Bologna process aiming to create a European Higher Education Area. A central tool in this work has been the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (also known as the ESG). The Twinning project offers an opportunity for applying the ESG in Azerbaijani higher education. One of the mandatory results of the project is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan (AzSG) in line with the ESG and to test them with three higher education institutions.

In April 2016 a Drafting Group was appointed by the Ministry of Education to work on a proposal for AzSG. A draft manual for the pilot evaluations was discussed with the Advisory Group in a seminar in June 2016. The draft was published on the Twinning project’s website in order to get feedback on the assessment areas and criteria. Amendments were made to the manual based on the feedback. The capacity of the pilot institutions to conduct a self-evaluation was supported through trainings and events in several previous missions. The self-evaluation reports were submitted by the three pilot universities in January-February 2017 and have been translated or are currently undergoing translation to English.  

The main aims of the pilot evaluations have been set as: supporting the strategic management of institutions, providing external feedback to the institutions’ own internal quality assurance procedures as well as informing the internal and external stakeholders of the compliance of the institutions’ quality assurance with the ESG. The pilot evaluations will have an institutional approach with the focus on teaching and learning. The evaluation report will provide the pilot institutions with information regarding their strengths and good practices as well as recommendations for the institutions’ further development.

3. Objectives and Tasks of the Mission

The overall aim of the Component 4 is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan, in line with the European Standards and Guidelines for Quality Assurance (ESG) and test them with three higher education institutions.

In Activity 4.5, the Standards and Guidelines are tested in pilot evaluations at three higher education institutions – Azerbaijan State Economic University, Azerbaijan State Pedagogical University and Azerbaijan Technical University. The objective of the first mission of Activity 4.5 was to prepare for the pilots by visiting the three pilot evaluation universities and meeting with the relevant staff to provide information on the arrangements of the pilots.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The meetings at the pilot evaluation universities - Azerbaijan State Economic University on February 21^st, at Azerbaijan State Pedagogical University on February 22^nd and at Azerbaijan Technical University on February 23^rd - were carried out successfully. The STE was accompanied by the MS Project Leader, the MS Junior Project Leader, staff from the Accreditation and Nostrification Office as well as the RTA’s team. The delegation were welcomed by the Rectors and Vice-Rectors of the universities, who assured their satisfaction with participating in the pilot evaluations. The meetings were attended by the staff involved in the self-evaluation process and the contact persons from each university. During the meetings, the STE reviewed the aims of the evaluation, the ethical principles (code of conduct) of the evaluation group, and provided information regarding the schedule, programme, interviews, composition of the evaluation group and the reporting process. The STE also informed the universities of what is expected from them in preparing and conducting the site visits. Discussions took place at each university regarding who should be invited to the interviews. The universities also had an opportunity to comment and ask questions regarding the evaluation process. Some of the questions included who to consider as external stakeholders for the university, and how syllabi are reviewed in the evaluation.

In addition, several organisational issues regarding the pilot evaluations were discussed and decided during the mission between the Project Leaders, STE, RTA’s office and in consultation with the MoE and ANO staff. These issues included the compensation for local participants in the evaluations groups, the attendance of observers and the training provided for the evaluation groups at the beginning of the site visit weeks.

6. Unexpected Results

An issue that rose during the mission was that the same student representative had been assigned to all three evaluation groups, causing an unfair workload for the student. The RTA’s office will try to recruit other students for two of the evaluation groups. 

7. Issues Left Open After the Mission

The participation of some of the local experts in the evaluation groups still needs to be confirmed, including the student representatives for the Azerbaijan State Economic University and the Azerbaijan Technical University.

8. Recommendations for Future Missions

The evaluation groups have a considerable amount of work ahead of them, and the role of the project managers as well as chairs is crucial. It is necessary to coordinate the process among the FINEEC and EKKA project managers, in order to implement the three pilot evaluations in a similar manner in all three universities.

9. Conclusions and General Remarks Concerning the Project

The mission provided a good opportunity to discuss the next steps in the pilot evaluations, and arrangements for the pilot evaluations are underway. The pilot evaluations will provide an excellent learning opportunity for all.

Mission report of Activity: 1.2 Revisions of the Legislative Framework

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 1. Legal and Regulatory

Activity: 1.2 Revisions of the Legislative Framework

Name of the Expert: Ms Heli Mattisen

Dates of the Mission: 20-24 February 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

As a result of the previous mission of Activity 1.2 that took place on 7 – 11 November 2016, the STEs Lagle Zobel and Helka Kekäläinen proposed a number of recommendations regarding the new draft of accreditation standards and accreditation rules developed by ANO. Based on the recommendations, ANO drafted new versions of standards and rules which was translated by Twinning office and sent to the STE’s of the Legal Component.

The final version of the Accreditation Rules of Higher Education Institutions (Accreditation Rules) and Standards determining the compliance of the activity of institution with the requirements of state education standards (Accreditation Standards) should be forwarded to the Cabinet of Ministers for approval in March 2017.

3. Objectives and Tasks of the Mission

The main objectives of the Mission were:

• Comprehensive discussion of ANO’s regulations in the light of the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) and identification of areas for improvement
• Formulation of concrete recommendations for the improvement of the drafts of Accreditation Standards and Accreditation Rules.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The expected results of the mission were achieved. The new draft of Accreditation Rules and Accreditation Standards, as well as recommendations were discussed. Due to the active participation of ANO, it was possible to critically evaluate the impact and applicability of recommendations.

Remarkable efforts have been made by ANO in reviewing the existing Accreditation Rules and Accreditation Standards.  In order to provide further support ANO regarding the improvement of their regulatory framework, we have the following recommendations:

Regarding the Accreditation Standards

• Some positive modifications regarding stakeholder involvement and use of student feedback in updating study programmes, the use of modern teaching materials and creative methods in teaching and learning, as well as a reference to ECTS have been made in the Accreditation Standards regarding part 4 (study programmes).

However, the concept of learning outcomes has not yet been introduced in the Standards. According to clause 1.2 of the ESG, institutions should have processes for the design and approval of their programs. The programs should be designed so that they meet the objectives set for them, including the intended learning outcomes. This should serve the overall objective of student-centred learning established in clause 1.3 of the ESG.

We strongly recommended adding to part 4 of the Draft Standard also explicit requirements for learning outcome based programme development as well as assessment of achieved learning outcomes. For example, following criteria could be added / reviewed:

• (Standard 4) Objectives, intended learning outcomes, admission and graduation requirements of the programmes are clearly defined; qualification resulting from the programme is clearly specified, communicated and referred to the appropriate level of the national qualifications framework (Remark: as the AzQF is not approved yet, the European QF or reviewed State Standards of Higher Education of the Republic of Azerbaijan could be referred to)
• (Standard 5)Practical work/internship supports the achievement of the learning outcomes of the programme.(in current draft: Institution shall organize internships for learners)
• (Standard 5) Student assessment shall be objective, consistent, transparent and support the achievement of learning outcomes.(in current draft: Correct, objective and transparent assessment of learners’ knowledge shall correspond to existing rules as well as to the international best practice)

• We also recommend reconsidering some criteria in the current draft version focused on checking the quantity rather than focusing on the quality of education, g. 2.8. The office work shall be run in line with the requirements of existing legislation; 3.2. Number of employees on the payroll (full-time, part-time etc.), as well as of those working on a substitution and hourly basis and their workload shall correspond to existing norms;

6.9. Number of doctoral and dissertation students assigned to one academic supervisor shall correspond to norms etc. The investment of expert-time requested for checking the fulfilment of the aforementioned requirements will not in balance with the level of potential impact for quality improvement.

• The inclusion of requirements for further sustainability of teaching staff cannot be detected in ANO regulations. It would still be advisable to add this requirement to the Accreditation Standards. The STEs recommended adding to the criteria for teaching staff also requirements for future sustainability. An example might be taken from the Accreditation Manual, which includes a following requirement: The distribution of full-time teaching staff by age and qualifications facilitates the sustainability of studies in a certain study area.

• Current drafts include the definition of accreditation introduced in Education Law. We recommend adding to the definition the purpose of improving the quality of higher education in Azerbaijan, as to ANO’s mission statement – Improve quality in higher education to increase local and international competitiveness of higher education.

Regarding the Accreditation Rules

• According to clause 3.1 of the revised ESG, the quality assurance activities undertaken by the agencies should have clear and explicit goals and objectives that are made available to the stakeholders. At the moment, the object of accreditation is not clear – is institutional accreditation combined with programme accreditation or should all programmes be accredited separately. Due to the voluminous amount of study programs, the current approach (accrediting the institutions together with study programs) does not allow devoting sufficient attention to the overall quality of the programs and thus, remains inevitably formalistic. We recommend clarifying the approach towards combining institutional and programme accreditation during the first period of accreditations in accreditation rules.

• As there is still no legal framework regarding the possible use of foreign agencies in the accreditation process, no further developments can be detected regarding the use of foreign agencies in the accreditation process in ANO regulations. Presumably, changes should first be made on legislative level. Thus, we strongly recommend including the conditions and procedure for the use of foreign agencies in the Statute of HEIs and mention this possibility also in the Accreditation Rules. Whether or not to officially recognize the results of any kind of cross-border evaluation is obviously a policy decision for the MoE, i.e. Accreditation Council headed by the Minister. However, we believe that allowing these kinds of activities would create a positive incentive for further internationalization of the HEIs and also ANO. It would also help to disseminate best practices of other agencies.

• According to clause 3.3 of the ESG, agencies should be independent and act autonomously. This includes organizational and operational independence and independence of formal outcomes. Currently, ANO enjoys a fair amount of operational independence. However, as ANO is a part of the MoE and the final decisions are made by the Accreditation Council chaired by the minister of education, organizational independence is clearly lacking.

As soon as the full trust of the Ministry has been gained, we recommend giving ANO full organizational and operational autonomy regarding its procedures and formal outcomes of the quality assurance processes. Implementing this recommendation definitely requires some fundamental changes on legislative level.

• As a remarkable development, changes have been made in clause 6.1 of the Accreditation Rules regarding the election of Accreditation Council on a permanent basis for four years. Although the inclusion of student members in the Council has not been specifically mentioned in the Rules, an explanatory statement has been added to the Rules (although it is not quite clear whether this will also remain in the final version of the Rules). It would still be advisable to make the inclusion of a student member in the Council obligatory and also specify the amount of members in the Council. We also recommend defining the proportion of independent experts in the Accreditation Council and agree upon the criteria for their election. An example of composition rules for the council applied by EKKA can be found here.

The reviewed principles of election and composition of the Council, including possible limitations, should be included also in the Statute of HEIs.

• According to clause 2.5 of the ESG, any outcomes or judgments made as the result of external quality assurance should be based on explicit and published criteria that are applied consistently, irrespective of whether the process leads to a formal decision. In other words, to ensure equal and fair treatment of the HEIs, the margin of discretion of the Accreditation Council should be limited by some basic decision-making principles.

A worksheet has been prepared for the Accreditation Commission (in the Final Review of the Accreditation Commission) for determining whether different accreditation areas are in line, partially in line or not in line with the Accreditation Standards.

The Accreditation Rules (see clauses 6.4 – 6.7) give the Accreditation Council an option to make one of three decisions based on the Accreditation Review: to fully accredit the institution/study programme if they meet the standards; to accredit the institution/study programme for 1 year if they partially meet the standards; and not to accredit the institution/study programme if the standards are not met.

However, the principles, pursuant to which the Council determines whether the standards are fully/partially/not met have not been elaborated in the Rules, providing the Council with a very wide margin of discretion in decision-making. In order to ensure equal treatment of all the HEIs and guarantee the consistency and predictability of accreditation results, it would be advisable to develop some sort of a framework for decision-making for the Council. E.g, it could be clarified  - either in the Accreditation Rules or some other ANO regulation made available to the public - how the assessments given by the Accreditation Commission to different assessment areas affect the final outcome of accreditation process.

• Pursuant to clause 2.4 of the ESG, external quality assurance should be carried out by groups of external experts that include (a) student member(s). As a very positive development in line with the ESG, the possibility of involving student members as well as foreign experts in the Commission have been added to clause 4.4 of the Accreditation Rules. However, according the ESG, the involvement of a student member in mandatory.

• According to clause 2.6 of the ESG, full reports by the experts should be published, clear and accessible to the academic community, external partners and other interested individuals. According to clause 6.8.1 of the reviewed version of the Accreditation Rules, the outcome of accreditation of the institution and its separate study programmes is posted on the official homepage of the Office. This is in line with the recommendation given by the STEs, but it should be clear that not only the accreditation decisions but also the accreditation reports should be available to the public.

• At the moment, follow-up activities have not been described in the ANO regulations. If ANO is planning to organize such activities, it would be advisable to consider elaborating this also in the regulations. According to clause 2.3 of the ESG, as a part of implementing process of external QA activities, agencies should have a consistent follow-up process for considering the action taken by the institution.

6. Unexpected Results

There were no unexpected results.

7. Issues Left Open After the Mission

There were no issues left open after the Mission.

8. Recommendations for Future Missions

There are 2 missions left in the Component 1 and it is important that the revision of the State Standards of Higher Education and the Statute of Higher Education Institution planned for the last mission week will and support the development of quality in higher education in Azerbaijan. 

9. Conclusions and General Remarks Concerning the Project

The sound and constructive cooperation between the higher education department and ANO in developing the legal framework for quality assurance in higher education is crucial.

Mission report of Activity: 3.5 Enhancing Practices and Procedures

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 3. Developing AzQF

Activity: 3.5 Enhancing Practices and Procedures

Name of the Expert: Mr Gunnar Vaht; Ms Baiba Ramina

Dates of the Mission: 27 February – 3 March 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

The Republic of Azerbaijan signed the Lisbon Recognition Convention (LRC) in 1997 and was first country that ratified (1998) the Convention. The preliminary system of recognition of foreign qualifications was introduced in 1992, but the Government Regulation on recognition process according to the principles of the LRC was adopted in 2003.

Until 2016 the competent authority for nostrification was Ministry of Education, since April 2016 the recognition and nostrification tasks were transferred to the ANO when this new institution was established for accreditation and nostrification/recognition.

The Republic of Azerbaijan is highly interested in qualified specialists studied abroad, and for this purpose there is a strict and detailed assessment of study programmes and foreign qualifications. The recognition authority often meets the problem that the quality of some individual qualifications from some countries of former Soviet Union is not at appropriate level to the labor market in Azerbaijan.

About 70% of foreign qualifications are recognised in practice, 20% are not recognised on the reason that knowledge of holders of foreign qualifications are at lower level than requirements in national standards and 10% of applicants present incomplete documentation for nostification.

The ANO is the only competent authority for recognition of foreign qualifications in the Republic of Azerbaijan. The holder of qualification should present the application together with all documentation on-line to the ANO. According to the national legislation, all documents must be translated and certified by apostille. In case the country is not party to the Hague Convention, the legalisation is required. The time limit is usually one month and the process is not free of charge to the applicant.

The procedure to the nostrification has three steps: 1) checking application and documents submitted for nostrification, and assessing whether the qualification is awarded by a recognized education institution; 2) assessing and giving equivalency to the qualification, and 3) inviting the applicant to the interview, and making final nostrification decision.

The assessment of foreign qualification for nostrification is based on seeking equivalency of the study programme. During the assessment of programme it is compared whether the programme completed abroad correspond to the national standard of the same or similar programme. The assessment of equivalency takes place at universities where the same or similar programmes are offered. The comparison of equivalency includes the list of courses and number of hours of each course. The programme content should be the same or very similar to the programme provided by the universities in Azerbaijan. In case the programme is not offered by the universities in Azerbaijan, the foreign qualification should be not recognized.

If a foreign qualification is not recognized, the ANO will explain the reason of rejection. The applicant has right to appeal the nostrification decision.

At the end of nostrification process an applicant receive two certificates – one for general recognition and one for nostrification.

3. Objectives and Tasks of the Mission

The objective of the mission was to facilitate the elaboration of a proposal for reflecting higher education of the AzQF in Azerbaijani recognition procedures and practices. The objective includes one-day national seminar on academic recognition of foreign qualifications and periods of study in Europe and Republic of Azerbaijan, and workshop on recognition practices and assessment criteria to the staff of ANO. The main task of the mission was to share information about best practices in Europe and to facilitate recognition criteria and procedure in the Republic of Azerbaijan.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The expected results of the mission were proposals for updating recognition procedures and practice.

All expected results were achieved. Seminar on assessment and recognition of foreign qualifications for the representatives of the Ministry of Education, ANO and higher education institutions, and workshop together with the staff of ANO were organised.

6. Unexpected Results

No unexpected results occurred.

7. Issues Left Open After the Mission

No issues regarding the mission were left open. The purpose of the mission was fulfilled.

8. Recommendations for Future Missions

No missions foreseen with the project.

9. Conclusions and General Remarks Concerning the Project

Since the ANO was established in 2016 there have been successful developments for assessment and recognition of foreign qualifications in the Republic of Azerbaijan. The staff of the recognition division is open to introduce new developments according to the LRC and its subsidiary texts.

The ANO receives a lot of foreign qualifications for recognition, the important amount of applicants are coming from the countries of former Soviet Union, mostly from Russian Federation, Ukraine and Georgia, where fake degrees and studies are problems to the office work. The ANO is putting a lot of efforts to check the authenticity of educational documents, and quite a lot of time is spent for verification and interviews of the holders of qualifications. Verification of evidence of studies is important part of recognition, however it could be clearly stated in operational charter of recognition, that interviews is not a tool for all applicants, and that it is used only when evaluator have some doubts about evidence of studies. Hereby it is suggested to create schematic outline as it is used in European Area of Recognition (EAR) Manual http://ear.enic-naric.net/emanual/ .

The workshop with the staff of ANO was focused on recognition criteria, and special cases and practices with the qualifications from different countries.

Current system of recognition of foreign qualifications is focusing on the content of the programme, the content of each course, number of working hours and other input elements that makes the recognition too detailed and complex. According to the national regulation: definition of the equivalence of higher education specialties is carried out through: Examination – recognition of higher education specialties of foreign countries and definition of compliance of the education acquired in the foreign country with educational standards of the Republic of Azerbaijan  https://www.nostrifikasiya.edu.az/frontend/en/content/item/9-%C3%BCmumi-qaydalar.

According to the subsidiary texts to the LRC the recognition of foreign qualifications should be based on five key elements of the qualifications – level, workload, profile, quality and learning outcomes, and the function of a qualification (formal right of the holder of qualification in home system of education). The ANO works for simplification of the assessment process, but the draft guidelines for new recognition system are not yet adopted by the Government. Therefore it can be suggested that the Ministry of Education will improve the recognition procedures moving from equivalence to the recognition unless substantial differences can be shown between the foreign and national qualification. It is suggested to implement the assessment and recognition system in the light of guidelines approved by the ENIC (European Network of Information Centres on Academic Recognition) – European Area of Recognition (EAR) manual, which is a practical tool for recognition authorities, credential evaluators and admission officers.

Also it can be suggested to use and implement subsidiary texts to the LRC to improve fair recognition of qualifications concerning higher education. Recommendation on Criteria and Procedures for Assessment of Foreign Qualifications (revised, adopted in 2010), and Recommendation on the Use the Qualifications Frameworks in the Recognition of Foreign Qualifications (adopted in 2013) are the fundamental tools for this process to introduce recommended criteria and procedure towards fair recognition based on learning outcomes.

According to the Recommendation on the Use the Qualifications Frameworks in the Recognition of Foreign Qualifications, the competent recognition authorities should also refer to the National Qualifications Framework, European Qualifications Frameworks and other similar Qualifications Frameworks as part of the assessment process. Therefore the development of Azerbaijan Qualification Framework will be a good transparency tool for the recognition of qualifications awarded by the education institutions in Azerbaijan.

LRC states that authorities shall provide adequate information on any institution belonging to its higher education system, and on any programme operated by these institutions. Higher education institutions in Azerbaijan are recommended to provide minimum information about their higher education programmes and qualification on their websites. Minimum information should be provided not only on national, but also on commonly used language, preferable in English.

It is very important to provide information about the educational system of Azerbaijan and on recognition issues on the website of ANO.

ENIC Network is useful in collecting information about education systems and lists of recognised education institutions. ANO looking forward to cooperate with the activities of the network, including the participation on recognition listserv (ENIC listserv). The ENIC network is established to exchange information and support assessment and recognition of foreign qualifications as well to exchange information on higher education systems and recognized institutions. Therefore a suggestion is also to improve information provision about Azerbaijani higher education system and recognition in the network´s website www.enic-naric.net.

Effort should be put to improve language skills of the recognition staff of ANO. The working language of ENIC Network is English, also all projects introduced by the European Commission and working documentation presented by the Council of Europe and UNESCO are in English. Capacity building of the staff of ANO should be part of development strategy of ANO. Therefore the suggestion is to organise regular staff training in the forms of study visits to other countries, on-line training courses, participation in conferences and seminars organised by the ENIC offices or international organisations in the field of recognition.

To improve cooperation with universities and disseminate information on recognition practices it is suggested to organize regular seminars on recognition for higher education audience.

ANO recognition terminology is recommended to use in light of Lisbon Recognition Convention. Therefore the use of term “nostricifation”, which is common term for old recognition approaches, at least on English translations is recommended to change to term “recognition”.

Higher education institutions have interest to develop internationalisation activities and therefore have interest on recognition procedures for admission of students holding a foreign qualification. Seminar for higher education institutions on recognition issues was held in Baku State University on 1 March 2017. There were a total of 34 participants mainly from the universities as Azerbaijan State Oil and Industry University, Azerbaijan State Arts and Culture University, Azerbaijan Technical University, Khazar University, Azerbaijan State Economic University, Azerbaijan University of Languages, Baku Engineering University, Sumgait State University, Nakhchivan State University, Azerbaijan State Pedagogical University, but also participants from Ministry of Education and ANO. The agenda of the seminar was focused on the presentations on international legal instruments; reforms in the European Higher Education Area; assessment recognition practices in Europa; assessment criteria and procedure; recognition networks; information provision; and recognition system, principles and procedure in the Republic of Azerbaijan.

Mission report of Activity: 4.5 Pilot Evaluations

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)
AZ/14/ENP/OT/31

EU Short Term Expert Mission Report

1. Basic Information

Component and Activity:

Component: 4. Standards and Guidelines for QA in HE

Activity: 4.5 Pilot Evaluations

Name of the Expert: Ms Eve Eisenschmidt, Mr Hannu Apajalahti, Ms Tiia Bach

Dates of the Mission: 10 – 14 April 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

International cooperation in quality assurance has been an essential element of the Bologna process aiming to create a European Higher Education Area. A central tool in this work has been the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (also known as the ESG).

The Twinning project offers an opportunity for applying the ESG in Azerbaijani higher education. One of the mandatory results of the project is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan (AzSG) in line with the ESG and to test them with three higher education institutions.

In April 2016 a Drafting Group was appointed by the Ministry of Education to work on a proposal for AzSG. A draft manual for the pilot evaluations was discussed with the Advisory Group in a seminar in June 2016. The draft was published on the Twinning project’s website in order to get feedback on the assessment areas and criteria. Amendments were made to the manual based on the feedback. The capacity of the pilot institutions to conduct a self-evaluation was supported through several trainings, which took place starting in September 2016. The self-evaluation reports were submitted by the three pilot universities in January-February 2017 and were translated into English. In February 2017 informative visits were conducted to all three higher education institutions participating in the pilot. Practical details of the upcoming site visits were discussed during these visits.
The main aims of the pilot evaluations have been set as: supporting the strategic management of institutions, providing external feedback to the institutions’ own internal quality assurance procedures as well as informing the internal and external stakeholders of the compliance of the institutions’ quality assurance with the ESG. The pilot evaluations will have an institutional approach with the focus on teaching and learning. The evaluation report will provide the pilot institutions with information regarding their strengths and good practices as well as recommendations for the institutions’ further development.

3. Objectives and Tasks of the Mission

The overall aim of the Component 4 is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan, in line with the European Standards and Guidelines for Quality Assurance (ESG). The objective of Activity 4.5 is to coordinate a trial/test run of the Standards and Guidelines at three higher education institutions in Azerbaijan. The first pilot was conducted at the Azerbaijan State Economic University during the STE mission of 3-7 April 2017.

The aim of the current mission was to conduct a pilot evaluation at Azerbaijan State Pedagogical University during the week of 10-14 April 2017. The first day of the mission focused on preparing the members of the evaluation group for the site visit. That followed by a three-day visit to the university, during which the evaluation team held interviews with the management, staff, students, alumni and external stakeholders of the institution. The outcome of the visit will be an evaluation report, which in addition to recommendations to the university under evaluation will also provide suggestions to the Ministry of Education regarding regulations governing the Azerbaijani higher education.

4. Time Schedule of the Mission

A detailed schedule of the mission is attached to the report (see Appendix 1).

5. Achievement of the Expected Results

The expected results for the mission were achieved: a practical testing of the Standards and Guidelines was carried out at Azerbaijan State Pedagogical University.

A long and thorough discussion about challenges and opportunities of Azerbaijan’s higher education was held among all participants during the kick-off meeting at the Ministry of Education on the first day of the mission. During the second half of the day the team had its meeting, which included a training for the team members by STEs Ms. Eisenschmidt (chair of the evaluation group) and Ms Bach (Project Coordinator) who reviewed the aims of the evaluation, the code of ethics guiding the evaluation team, and how to conduct the interviews and present questions. The team members then focused on formulating detailed questions for the interviews that were going to be held during the site visit with various stakeholders and developing the interview strategy.

The site visit was carried out successfully over three days and the team interviewed altogether nearly 80 members of the university on one of the campuses in Baku. Interview sessions were held with the management, deans, teaching staff, support and administrative staff, students, alumni and employers. During the interviews, the evaluation team was able to ask questions to verify and supplement observations that had been made on the basis of the self-evaluation report and other material submitted by the university. The interviews were conducted in English and translated into Azerbaijani. The interviews were held in a good and open atmosphere.
Three observers – one from Baku State University and two from the Accreditation and Nostrification Office (ANO) – accompanied the evaluation team in order to learn from such evaluation process.
The evaluation team had a good cooperation with the contact person and other representatives of the university who were extremely helpful during the entire site visit by providing everything that was necessary for the team to successfully complete its work.

6. Unexpected Results

None.

7. Issues Left Open After the Mission

None.

8. Recommendations for Future Missions

As recommendations for the next pilot evaluation, the STEs suggest the following:

- A detailed schedule of the site visit was sent only a day before arrival at the university. Thus, the evaluation group did not have a clear idea beforehand who exactly (people and their positions) they were going to meet. If possible, ensure that the team receives it earlier to better prepare for the interviews.

- It is highly recommended that even if the evaluation process is new for the local team members, all members do the preparation work on the Self-Evaluation Report of the institution under evaluation and give their initial input by sending to the Project Manager (STE from FINEEC/EKKA) questions that arise while reading the report, and list the topics that should be further discussed during the site visit. This would allow the Project Manager to do considerable amount of preparatory work by the first team meeting on Monday (collect the questions, sort them, etc.).

- It is important to allow ample time before the site visit for the team members to get to know each other better, and for the STEs take that opportunity to get to know the local context, culture, what kind of behaviour might be appropriate during the visit and what is not.

- The team leader should pay enough attention that everyone from the team has had enough time to share his/her own understanding of the situation before the site visit. Some extra time might be needed to figure out the best possible approach to asking the questions and dividing the roles among the team members. It is important to secure that each team member is actively involved in the whole process.

- The project manager (STE from FINEEC/EKKA) should use enough time to establish a shared understanding of the practical details (especially number of interviewees, interview settings, room arrangements, name labels for the interviews, and rules during the interviews) with the contact person of the university. If necessary, make re-arrangements during the site visit (e.g., if the rectorate has been prepared for the interviews, ask for a more appropriate, separate room where discussions can be held). In that sense the meeting with the university’s contact person was extremely helpful on the first day of the mission; the team could go through the practical details of the visit (some of which seemed to come as a surprise, although this had been shared by email earlier), clarify some issues and questions, request for additional materials, etc.

9. Conclusions and General Remarks Concerning the Project

The pilot evaluations are a pivotal point in the Twinning project. Two pilot evaluations have now been conducted. The pilot evaluation in Azerbaijan State Pedagogical University was conducted in good cooperation with the participating higher education institution. ANO has been actively involved in the process and these evaluations have been hopefully an interesting experience for the local observers. The evaluation team felt very well supported and taken care of by the Resident Twinning Adviser and other support staff during the whole week. We are equally thankful for excellent interpretation that was provided for the interviews during the site visit.

The Pilot Analysis has been planned for early June. Although the timeline for the pilots is quite short and the teams are pressed for time, they will likely provide plenty of useful feedback on the criteria and evaluation model of the Standards and Guidelines for Quality Assurance of Higher Education in Azerbaijan.

Mission report of Activity: 4.5 Pilot Evaluations

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 4. Standards and Guidelines for QA in HE

Activity: 4.5 Pilot Evaluations

Name of the Expert: Ms. Kristiina Tõnnisson, Mr. Mikko Vieltojärvi, Ms. Hilla Aurén

Dates of the Mission: 3 – 7 April 2017 (Ms. Kristiina Tõnnisson), 5 – 7 April 2017 (Ms. Hilla Aurén and Mr. Mikko Vieltojärvi)

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

International cooperation in quality assurance has been an essential element of the Bologna process aiming to create a European Higher Education Area. A central tool in this work has been the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (also known as the ESG). The Twinning project offers an opportunity for applying the ESG in Azerbaijani higher education. One of the mandatory results of the project is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan (AzSG) in line with the ESG and to test them with three higher education institutions.

In April 2016 a Drafting Group was appointed by the Ministry of Education to work on a proposal for AzSG. A draft manual for the pilot evaluations was discussed with the Advisory Group in a seminar in June 2016. The draft was published on the Twinning project’s website in order to get feedback on the assessment areas and criteria. Amendments were made to the manual based on the feedback. The capacity of the pilot institutions to conduct a self-evaluation was supported through several trainings, which took place starting in September 2016. The self-evaluation reports were submitted by the three pilot universities in January-February 2017 and were translated to English. In February 2017 informative visits were conducted to all three higher education institutions participating in the pilot. Practical details of the upcoming site visits were discussed during these visits.

The main aims of the pilot evaluations have been set as: supporting the strategic management of institutions, providing external feedback to the institutions’ own internal quality assurance procedures as well as informing the internal and external stakeholders of the compliance of the institutions’ quality assurance with the ESG. The pilot evaluations will have an institutional approach with the focus on teaching and learning. The evaluation report will provide the pilot institutions with information regarding their strengths and good practices as well as recommendations for the institutions’ further development.

3. Objectives and Tasks of the Mission

The overall aim of the Component 4 is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan, in line with the European Standards and Guidelines for Quality Assurance (ESG) and test them with three higher education institutions.

The objective of Activity 4.5 is to coordinate a trial/test run of the Standards and Guidelines at three higher education institutions in Azerbaijan. The first pilot was conducted at the Azerbaijan State Economic University during the STE mission of 3-7 April 2017. The first two days of the mission focused on preparing the experts of the evaluation group for the site visit. During the last three days of the mission, the evaluation group conducted a site visit to the University, interviewing management, staff, and students of the University.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The expected results for the mission were achieved: A practical testing of the Standards and Guidelines was carried out at the Azerbaijan State Economic University.

During the training, STE Ms. Tõnnisson reviewed the aims of the evaluation, the code of ethics guiding the evaluation team, and how to conduct the interviews and present questions. A long and thorough discussion about challenges and opportunities of Azerbaijan’s higher education was held among all participants during the kick-off meeting. Later the discussion focused on challenges of the site visits and possibilities of writing evaluation report. The second day focused more concretely on preparing the meeting with various stakeholders and on formulating the questions. The team worked the full day on forming and reforming, on structuring and restructuring, on focusing and refocusing the questions for all sessions for the meetings during the 3 days long site visit.

The site visit was carried out successfully over three days and the team interviewed altogether nearly 100 members of the university on three different campuses. Interview sessions were held with the management, teaching staff, support and administrative staff as well as students from different faculties. During the interviews, the evaluation team was able to ask questions to verify and supplement observations made based on the self-evaluation report and other material submitted by the university.

Observers from the Accreditation and Nostrification Office (ANO) accompanied the evaluation team. The interviews were conducted in English, translated into Azerbaijani.

The University contact person had expressed beforehand concerns about external stakeholders being unable to attend the interviews. The evaluation team used the time that was allocated for interviewing external stakeholders by interviewing personnel from the Career Centre, the Internship Centre and the Communication and Marketing Department. The ability of the evaluation team to quickly adapt to changing conditions during the interview process was a key to making the site visit successful. The cooperation with the University was carried out in a good atmosphere.

6. Unexpected Results

The site visit had to be rescheduled with very short notice, and as a result the original mission plan was adjusted. The evaluation team ended up having one additional day for an all-day meeting among the evaluation team, which provided very useful. Under the guidance and instruction of STE Kristiina Tõnnisson, the team was able to create a common understanding of the most important issues for the interviews and to prepare the final list of interview questions. Although this day was not part of the original mission plan, due to the other two STEs’ difficulties in obtaining visas during the Nowroz holiday, the site visit was moved toward the end of the week (Wednesday-Friday) and this opened up one more day for the team to prepare.

7. Issues Left Open After the Mission

The request of compensation for local participants of the evaluation team is being processed at the Ministry of Education. In addition, the participation of the evaluation team members in the Pilot Analysis has to be confirmed. 

8. Recommendations for Future Missions

As recommendations for the next two pilot evaluations, the STEs suggest the following:

• Reserve considerable amount of time for creating a common understanding among the evaluation team before the interviews. It is important that team members are prepared and they have thought about what are the main issues to find out during interviews, since time in-country before the site visit will be limited.
• The project manager (STE from FINEEC/EKKA) should use enough time to establish a shared understanding of the practical details (especially number of interviewees, interview settings, room arrangements and rules during the interviews) with the contact person of the university.
• There are several terms which the STEs realized have different meanings or lack meaning in the local context, and it is crucial to establish shared understanding of such concepts (e.g. course, syllabus, curricula, study programme, subject, learning outcome etc.); It would be useful to have some short written introduction to national higher education system before the visit.
• The team leader should pay enough attention that everyone from the team has had enough time to share his/her own understanding of the situation before the site visit. Some extra time might be needed to figure out the best possible approach to asking the questions and dividing the roles among the team members. It is important to secure that each team member is actively involved in the whole process.

9. Conclusions and General Remarks Concerning the Project

The Pilot Evaluations are a pivotal point in the Twinning project. One Pilot Evaluation has now been conducted and two more are to take place within the month of April. The Pilot was conducted in good cooperation with the participating higher education institution. ANO has also been actively involved and will be sending observers to all three site visits.

The Pilot Analysis is set to take place in early June. Although the timeline for the Pilots is quite short and the teams are pressed for time, the Pilots will likely provide plenty of useful feedback on the criteria and evaluation model of the Standards and Guidelines for Quality Assurance of Higher Education in Azerbaijan.

Mission report of Activity: 4.5 Pilot Evaluations

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 4. Standards and Guidelines for QA in HE

Activity: 4.5 Pilot Evaluations

Name of the Experts: Mr. Heikki Malinen, Mr. Tauno Otto, Mr. Touko Apajalahti

Dates of the Mission: 24 – 28 April 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

International cooperation in quality assurance has been an essential element of the Bologna process aiming to create a European Higher Education Area. A central tool in this work has been the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (also known as the ESG). The Twinning project offers an opportunity for applying the ESG in Azerbaijani higher education. One of the mandatory results of the project is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan (AzSG) in line with the ESG and to test them with three higher education institutions.

In April 2016 a Drafting Group was appointed by the Ministry of Education to work on a proposal for AzSG. A draft manual for the pilot evaluations was discussed with the Advisory Group in a seminar in June 2016. The draft was published on the Twinning project’s website in order to get feedback on the assessment areas and criteria. Amendments were made to the manual based on the feedback. The capacity of the pilot institutions to conduct a self-evaluation was supported through several trainings, which took place starting in September 2016. The self-evaluation reports were submitted by the three pilot universities in January-February 2017 and were translated to English. In February 2017 informative visits were conducted to all three higher education institutions participating in the pilot. Practical details of the upcoming site visits were discussed during these visits.

The main aims of the pilot evaluations have been set as: supporting the strategic management of institutions, providing external feedback to the institutions’ own internal quality assurance procedures as well as informing the internal and external stakeholders of the compliance of the institutions’ quality assurance with the ESG. The pilot evaluations will have an institutional approach with the focus on teaching and learning. The evaluation report will provide the pilot institutions with information regarding their strengths and good practices as well as recommendations for the institutions’ further development.

3. Objectives and Tasks of the Mission

The overall aim of the Component 4 is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan, in line with the European Standards and Guidelines for Quality Assurance (ESG) and test them with three higher education institutions.

The objective of Activity 4.5 is to coordinate a trial/test run of the Standards and Guidelines at three higher education institutions in Azerbaijan. The objective of this mission was to conduct the third pilot at the Azerbaijan Technical University during the STE mission of 24-28 April 2017.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The expected results for the mission were achieved: A practical testing of the Standards and Guidelines was carried out at the Azerbaijan Technical University.

The first day of the mission focused on preparing the experts of the evaluation group for the site visit. During the following three days of the mission, the evaluation group conducted a site visit to the University, interviewing management, staff, and students of the University. On the last day, the evaluation team began the drafting of the review report.

During the first day, first a kick-off meeting was organised at the Ministry of Education in order to discuss the Azerbaijani higher education system with the evaluation team.

In the afternoon’s meeting of the evaluation team, STEs reviewed together with the rest of the team  the aims of the evaluation, the code of ethics guiding the evaluation team, and how to conduct the interviews and present questions. Also, during the meeting the team finalised preparations for the meetings with various stakeholders during the site visit.

The site visit was carried out successfully over three days and the team interviewed altogether nearly 100 members of the university community. Interview sessions were held with the management, teaching staff, support and administrative staff as well as students from different faculties, graduates and external stakeholders. During the interviews, the evaluation team discussed various topics with the interviewees to verify and supplement observations made based on the self-evaluation report and other material submitted by the university.

Observers from the Accreditation and Nostrification Office (ANO) accompanied the evaluation team. The interviews were conducted in English, translated into Azerbaijani.

6. Unexpected Results

No unexpected results were encountered.

7. Issues Left Open After the Mission

No issues were left open.

8. Recommendations for Future Missions

9. Conclusions and General Remarks Concerning the Project

All pilot evaluations have now been conducted. Also the third pilot was conducted in good cooperation with the participating university. ANO has also been actively involved in all three pilot evaluations.

It is recommended that ANO and other project participants will take advantage of the experience and of the upcoming analysis week in order to further building of the capacity of the universities to undertake such evaluations. Special attention should be paid into the self-evaluation phase, in order to have a comprehensive self-evaluation and to include all the necessary annexes. Also, it would be good to have more systematic training for the evaluators in an early phase of the evaluation process: both for those new team members for whom it is the first experience of such process, and for international members who do not have deep knowledge of the local higher education system before the process. However, in conclusion, based on the third pilot, the university and the local experts are fully capable of carrying out such processes in a fruitful and beneficial manner.

Mission report of Activity: 1.2 Revisions of the Legislative Framework

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 1. Legal and Regulatory

Activity: 1.2 Revisions of the Legislative Framework

Name of the Experts: Ms Sille Uusna, Ms Heli Mattisen

Dates of the Mission: 1-5 May 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

During the first missions of Activity 1.2, in 2016 the STEs gave to MoE some concrete recommendations regarding the new State standard and program for higher education (State standard) draft, in order to ensure its compatibility with the principles of the European higher education area and create consistency between different regulations concerning the HE subject area. The next 1.2. mission was focused on the Statute of higher education institutions (Statute of HEIs) and led to recommendations. The third mission of 1.2. concentrated to accreditation rules.

Recently, the MoE introduced to STE’s updated drafts of:

• the State Standard for HE;
• the Accreditation standards for evaluating the activities of HEIs;
• the Accreditation rules of HEIs and study programs.

3. Objectives and Tasks of the Mission

This mission purpose was to evaluate the progress that have been made by implementing STEs recommendations and to provide concrete input into legislative drafts based on the recommendations and to support the elaboration process of the legislation relevant to QA and the AzQF sections relevant for higher education, and adherence to the EHEA.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The expected results of the mission were achieved. In workshops, the recommendations were comprehensively discussed.

6. Unexpected Results

There were no unexpected results.

7. Issues Left Open After the Mission

According to the representatives of the MoE, the new draft of the Statues of HEIs is under preparation. The new draft should consider most of the recommendations made previously by the STEs. The MoE continues work with the draft of State standard.

8. Recommendations for Future Missions

The cooperation between experts from different departments in MoE should be fully recognized and strongly encouraged also during future Missions. State standards about doctoral education and learning outcomes and general quality requirements.

9. Conclusions and General Remarks Concerning the Project

Considerable efforts have been made by the Ministry of Education (MoE) in adopting the principles of European higher education area. The draft of the new State standard takes into account several recommendations that STEs have made during previous missions. However, the STEs still have recommendations which include some unevitable ones regarding the overall drafting process and further developments of the Statute of HEIs and the State standard, and accreditation rules and standards.

We were impressed by improvements made in both accreditation documents developed by ANO - Accreditation standards for evaluating the activities of higher education institutions and Accreditation rules of higher education institutions and study programs. The document providing the standards for accreditation is in sound accordance with ESG while taking into account national context. Pilote evaluations conducted in the framework of Twinning project will certainly give an evidence based input for further development of both documents.

URGENT RECOMMENDATIONS

Cooperation

1. During the mission, STEs have met with highly competent and motivated experts from HE Department, ANO and Legal Department. We strongly recommend promoting cooperation between MoE experts to combine competence in different fields. It is also advisable to use the format of joint brainstorming, working groups and also more informal communication during working process for discussing future initiatives of HE issues the MoE. Joint effort and promoted cooperation, also in other manner than formal feedback in official consulting process, could lead to an excellent result.

2. It would be most advisable to promote cooperation with stakeholders in HE field. For example, supporting the establishment of Rectors Conference or form a higher education advisory board, led by the MoE.

Hierarchy of legislation

3. At the moment there seems to be indistinctness about the hierarchy of different legal acts and provisions. We recommend that MoE experts from different departments could in a form of joint brainstorming draft a possible structure and hierarchy of reasonable HE legislation system, aiming to avoid overlapping while assuring coherence.

4. Some efforts have been made in reviewing the existing legislation and repealing the provisions not needed anymore (consolidation of accreditation regulations, a new draft for Scientific Board etc). We strongly recommend continuing with that kind of activities. For example, the separate regulations for each study level (BA, MA, PhD) should be consolidated in the State standard.

5. According to the representatives of MoE, a new draft of Statutes of HEIs will be prepared. The STEs recommend to develop the Statutes of HEIs to comprehensive higher education (HE) framework regulation which includes all basic principles of HE.

Quality requirements

6. At the moment, the quality requirements for establishing new HEIs/study programs have not been included in any general framework regulation yet. The link between opening a new study program and accreditation seems to be not clear.

In order to ensure the continuity, the quality requirements for opening new study program should be in accordance with the requirements later taken into account for accreditation. There may be a need for more detailed regulation how to initially evaluate the quality of a new curricula. The STEs recommend to use as much as possible the competence and recourses of ANO to provide continuity between initial and following evaluations.

We recommend:

Add to the Statute for HEI (framework act)

• Basic principles for opening a new curriculum, including initial quality control;
• Basic principles for accreditation, including the object of accreditation (institutions or study programs or both); accreditation period; accrediting body and it’s general formation principles, composition (incl. limitations); consequences of a negative accreditation.

Add to the State Standard of HE

• Basic quality standards describing in general terms the societal expectations towards the quality of HE in Azerbaijan. Based on the general quality standards ANO will specify the quality requirements and assessment criteria at the next level.

An example of general quality standards which are in line with ESG and could be the bases for more detailed requirements and criteria in the framework of external assessment:

educational values

• The higher education institution shall develop students’ ability to adapt to modern requirements and conditions, to be competitive, independent and creative individuals and responsible citizens, to live and act in information society and to build communication.
• The institution shall support academic integrity and is vigilant against academic fraud; it supports guarding against intolerance of any kind or discrimination against the students and staff.

learning outcomes and ECTS

• The design and development of study plans is in line with areas of activity and educational objectives of the institution, expectations of the society, labour market needs, and the latest research.
• Objectives, intended learning outcomes, admission and graduation requirements of the programs are clearly defined.

• Expected student workload is defined in credit point system at all levels of higher education.
• Qualification resulting from the program is referred to the appropriate level of the European qualifications framework in order to assure the comparability of qualifications.

academic staff

• Academic staff with adequate qualifications ensure the achievement of the objectives and learning outcomes of the study program, as well as the quality and sustainability of teaching and learning.
• The distribution of full-time teaching staff by age and qualifications facilitates the sustainability of studies.

teaching and learning process

• Teaching and learning takes into account the educational needs of students, provides flexible learning opportunities and encourages active participation of students in creating the learning process.
• Practical work/internship supports the achievement of the learning outcomes of the program.
• Student assessment is objective, consistent, transparent and supports the achievement of learning outcomes.
• Up-to-date teaching materials, innovative, creative and interactive learning methods and educational technology are used in teaching and learning process.

resources and support services

• Resources (teaching and learning environments, teaching materials, teaching aids and equipment, premises, financial resources) support the achievement of objectives in the study program.
• Students are provided academic, career and psychological counselling by the higher education institution. Students’ academic progress is monitored and supported; the institution has processes and tools to support special educational needs.
• Information technology tools for studies and research and development activities and connections to domestic and international information networks are accessible and necessary data bases are available.

Learning outcomes

7. The STEs recommend to include the HE level descriptors (levels 6-8) from the draft of NQF in an Annex to the State Standards for HE and to refer to their correspondence with a respective level in EQF. It gives a clear reference point to curriculum developers and supports Azeri HE to meet the objectives of the Bologna process.

8. As there is no differentiation of learning outcomes at level 8 between PhD and Doctor of Sciences, and the compulsory credit point system is not applicable for the program of Doctor of Sciences, therefore we recommend not to describe the DSc program in the State standards for higher education.

9. The principles of the student assessment are very well described (e.g., transparency, objectivity, fairness, self-evaluation, diverse methods and tools of assessment). In addition, the main aim of assessment should also be clearly stated: to measure whether the learning outcomes have been achieved as well as to support students’ learning. The assessment methods and criteria should be in full accordance with the learning outcome(s).

Accreditation rules and standards

10. According to clause 3.3 of the ESG, agencies should be independent and act autonomously. This includes organizational and operational independence and independence of formal outcomes. Currently, ANO enjoys a fair amount of operational independence. However, as ANO is a part of the MoE and the final decisions are made by the Accreditation Council chaired by the minister of education, organizational independence is clearly lacking.

As soon as the full trust of the Ministry has been gained, we recommend giving ANO full organizational and operational autonomy regarding its procedures and formal outcomes of the quality assurance processes. Implementing this recommendation definitely requires some fundamental changes on legislative level, but is essential.

11. The rules for the composition of the Accreditation Council have been removed from the last draft of the Accreditation Rules and Procedures, as they will be defined in another document Regulation of the Accreditation Council approved by the Ministry of Education. However, it is important to clarify the rules and principles (incl the number of Council members and the election period) for selection of the member of the Accreditation Council and make the inclusion of a student member in the Council obligatory. An example of composition rules for the council applied by EKKA can be found here.

12. According to clause 3.5. of ESG agencies should have adequate and appropriate resources, both human and financial, to carry out their work. It is in the public interest that agencies are adequately and appropriately funded, given higher education’s important impact on the development of societies and individuals. Therefore, it is essential to provide financial resources needed for conducting quality assurance activities in an objective, effective and efficient manner and for involving best experts.

There are different practices regarding financing of external quality assurance activities. The most common practice is, that the basic funding of agencies (covering permanent staff, premises and other permanent costs) is offered by the state and institutions are paying fees covering variable costs connected to the single accreditation procedures (expert fees, transport, accommodation). While ANO’s permanent costs are fairly covered by the MoE sources for covering variable costs depending on the scope of accreditations are clearly lacking. We recommend to develop and approve the accreditation fees for higher education institutions depending on the scope (size of the institution, number of programs etc) of the accreditation procedure.

13. Highly qualified and trained experts are the key actors in the accreditation procedure. In addition to their academic competence they have to be aware of their role in the procedure, follow the code of ethics of experts and have a common understanding of the quality standards (see also ESG 2.4). Therefore, it is essential to add to the requirements for a member of an expert group the participation in a training offered by ANO.

14. According to the clause 2.7. of ESG complaints and appeals processes should be clearly defined as part of the design of external quality assurance processes and communicated to the institutions.

A complaints procedure allows an institution to state its dissatisfaction about the conduct of the process or those carrying it out. In an appeals procedure, the institution questions the formal outcomes of the process, where it can demonstrate that the outcome is not based on sound evidence, that criteria have not been correctly applied or that the processes have not been consistently implemented. We recommend to include the description of the complaints and appeal procedures in the Accreditation Rules of Higher Education Institutions and Study Programs.

ADDITIONALLY, TO CONSIDER OTHER RECOMMENDATIONS PROVIDED IN PREVIOUS REPORTS

15. As there is still no legal framework regarding the possible use of foreign agencies in the accreditation process, no further developments can be detected regarding the use of foreign agencies in the accreditation process in ANO regulations. Presumably, changes should first be made on legislative level. Thus, we strongly recommend including the conditions and procedure for the use of foreign agencies in the Statute of HEIs and mention this possibility also in the Accreditation Rules. Whether or not to officially recognize the results of any kind of cross-border evaluation is obviously a policy decision for the MoE, i.e. Accreditation Council headed by the Minister. However, we believe that allowing these kinds of activities would create a positive incentive for further internationalization of the HEIs and also ANO. It would also help to disseminate best practices of other agencies.

16. The STEs, taking into account the recommendations of previous missions, point out the need to clarify in framework act the main division of responsibilities between Rector and Scientific Board. The STEs also recommend to consider some performance indicators to universities and to regulate the structure of self-administration of student unions in a centralized manner. Also, the STEs recommend to include the basic criteria and characteristics of a research university in either the Education Law or the Statute of HEIs. 

Mission report of Activity: 3.4 Further Implementation of the AzQF

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 3. Developing AzQF

Activity: 3.4 Further Implementation of the AzQF

Name of the Expert: Mr Olav Aarna, Ms Maiki Udam, Ms Aune Valk

Dates of the Mission: 8-12 May 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

The development of Azerbaijani National Qualifications Framework (AzQF) for lifelong learning started already in 2011 supported by the World Bank and the European Training Foundation (ETF) projects. By 2012, the draft Decree on establishing the AzQF was developed, stipulating the main principles, institutions involved, level descriptors, placement of types of qualifications, and quality assurance of qualifications. In 2012 the draft AzQF was submitted to the Council of Europe for expertise. Recommendations proposed by the Council of Europe expert have been taken into account while designing the AzQF final draft. Seminars introducing the AzQF to stakeholders and international partners were held in 2012 and 2014. Until recently, the Decree has still been a draft form.

In 2014, ETF prepared a comprehensive document “Analysis of existing qualification standards in Azerbaijan”, containing the descriptions of different qualifications' levels as well as concrete recommendations for further actions for 2015-2017 with division of roles and responsibilities in development, revision, approval and application of AzQF. A draft of the implementation plan for the AzQF has also been prepared by the ETF and Azerbaijani counterparts.

In November 2015 the Minister of Education introduced the draft AzQF document to the line ministries and other stakeholders. Feedback from different ministries included several recommendations for amendments in the text.

During the last two years the Ministry of Education (MoE) staff has been renewed fundamentally. Only one person, Mr. Azad Ahundov, has been involved in the development of AzQF from the very beginning. As a result, the new staff members, incl. the heads of departments and units of the MoE have limited understanding about the AzQF purpose, objectives, guiding principles, and the way to implement it. More importantly, until recently there has been no clear ownership of the AzQF in the MoE.

The current state of affairs is the following. In the first week of May 2017 the Minister of Education submitted the draft Decree on establishing the AzQF with minor changes to the Cabinet of Ministers for approval. Following the decisions taken at the meeting with the minister on 22 July 2016, a secretariat headed by the deputy head of Apparatus Mr. Farzali Gadirov has been established at the MoE to administer further development and implementation of the AzQF. National expert Mr. Ilkin Nazarov has also prepared a draft “Action Plan on Implementation of National Qualifications Framework for Lifelong Learning of the Republic of Azerbaijan (2018-2025)”, and a “Concept of establishment of National Commission and Secretariat for implementation of AzQF” commissioned by the MoE.

3. Objectives and Tasks of the Mission

Objective:

The elaboration of a roadmap for the further implementation of the AzQF sections relevant for higher education (leading towards a successful self-certification).

Tasks (methods):

• An one-day workshop on further implementation of the AzQF HE sections for the representatives of MoE and universities has been conducted for 10-20 people;
• A roadmap with objectives, responsibilities, timelines and an indication of required resources for the self-certification process has been drafted.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

All expected results were achieved.

6. Unexpected Results

There were no unexpected results.

7. Issues Left Open After the Mission

There are no issues that have been left open after this mission.

8. Recommendations for Future Missions

As it has been the final mission week of the Component 3 then there are no more missions coming.

For future projects we would recommend to deal especially with the following issues concerning the implementation of the AzQF:

• Developing the system for recognition of prior (including non-formal and informal) learning;
• Creating databases/IT system(s) for managing the information about students, staff, study progress, admission, graduation etc;
• Training teachers in innovative teaching and assessment methods.

9. Conclusions and General Remarks Concerning the Project

The mission has proved to be successful. The AzQF development and implementation process has gained new momentum. The draft “Action Plan on Implementation of National Qualifications Framework for Lifelong Learning of the Republic of Azerbaijan (2018-2025)”, and the “Concept of establishment of National Commission and Secretariat for implementation of AzQF” are a good starting point for launching the AzQF implementation.

A general recommendation would be to add also wider impact indicators to the action plan, e.g., increased mobility of students and staff, increased participation of adult people in lifelong learning, employment rate of graduates of VET and higher education, etc.

The recommendations concerning further development of the Action Plan are incorporated into the text of the document and in the column under the title Note (cf. Annex 2).

Discussions with the employees of the MoE and participants of the seminar on 11 May 2017 revealed the following overarching problem areas requiring special attention for the success of the AzQF implementation process:

• Timeframe of implementation

Implementation of the AzQF means developing a new and contemporary National Qualifications System, with the AzQF serving as a nucleus of this system. The Action Plan encompasses the period 2018-2025. It foresees actions in five groups:

1. Improving legislative framework in connection with implementation of AzQF.
2. Creating institutional system coordinating AzQF implementation.
3. Bringing content of qualifications into conformity with the labor market requirements.
4. Improving and strengthening quality assurance system.
5. Promoting implementation of result-based qualification system.

Many activities cover very long period, e.g. 4.1. Creating National Qualifications Register (2018-2025).  Obviously the creation of the Register does not take seven years. More important is to know by which time the Regiser will become functional.

We recommend to define intermediate milestones of the AzQF implementation, e.g. self-certification of the AzQF in reference to the Qualifications Framework of the European Higher Education Area (and European Qualifications Framework) by 2020 (p.5.8).

• Institutional structure for the AzQF implementation

The Concept foresees establishing two new bodies for coordinating the implementation process:

• National Commission,

The Secretariat is envisaged as an independent structural unit of the MoE, performing three functions:

• Management of qualifications;
• Development of qualification standards;
• Development of the National Qualifications Register.

The functions and structure of the Secretariat seem appropriate. However, performing these functions is a full-time job for substantial number of employees. Therefore, the experts strongly recommend:

• Establish the Secretariat as an independent department of the MoE with permanent staff (gradually growing in number);
• Consider changing the name for more appropriate one, because the present one is misleading. It rather refers to the secretarial support for the National Commission.
• Formalise the roles of the institutional partners involved in the different aspects of the AzQF implementation, incl. functions performed for different types of qualifications, on an appropriate level.

It is of utmost importance to engage all partners and stakeholders (ministries and agencies, educational institutions, student unions, employers’ organisations) into the implementation process of the AzQF. It requires regular communication not only in formal setting of meetings of the National Commission and other decision making bodies, but also different less formal means of informing each other about the developments and problems with respect to the AzQF implementation.

• Qualifications standards

In the draft Action Plan on Implementation of National Qualifications Framework for Lifelong Learning of the Republic of Azerbaijan (2018-2025) development of different qualification standards is foreseen (see 3.2, 3.5, 3.6). We recommend to have two sets of standards:

• Level standards for eight levels of the AzQF;
• Sectoral standards for specific occupations/professions, including tasks and competences of a particular occupation/profession. These standards could also include assessment standards for assessing, validating and certification persons’ competence.

Based on the level standards and sectoral standards both the national and institutional curricula can be developed. There is no need for separate qualification standards on formal education (3.6.1) and qualification standards for non-formal and informal education (3.6.2). 

• Human resource development: training and networking

It is obvious that engagement of people, training and supporting them is crucial in the successful application of any new initiative.

Besides classical training that is necessary for several groups and topics, we recommend to create and support networks of practitioners. Professional networks have proved to be better teacher training method than traditional face to face training (see additional Measure 2.7).

For developing and applying outcomes-based curricula, introducing new teaching and assessment methods, developing systems for recognition of prior (informal and non-formal) learning

(counselling, assessment and accreditation), supporting mobility and academic recognition, developing lifelong learning opportunities at universities etc, there must be people responsible for these topics in the HEIs. In the starting phase, there is often just one person in the institution who shall introduce it. For him/her it is very difficult if there is no support. The best supporters are people with similar tasks and experience in other institutions. Helping these people to get together and create a network is very important if you want that the best practices are shared. Centralised leading and support system may be not enough in these cases.

As the largest target group in applying the changes are university teachers, it would be useful to set up centre(s)/units that are responsible for teaching academic staff (see additional Measure 2.6).

• Recognition of prior learning

Recognition of prior (informal and non-formal) learning (RPL) is a corner stone of lifelong learning mindset. Currently this topic is covered at two levels in the action plan: legislation and promotion to end-users. The most difficult target group is, however, universities. Preparing their systems and training their people (counselling and assessment of RPL) are even more crucial and recommended to add to the action plan. 

• Quality Assurance

Quality assurance, both internal and external, is an important part of national qualifications systems. One objective in the action plan is Improving internal and external quality assurance system and linking it with ESG and EQAVET (4.3).

The topics that need further consideration with respect to the ESG are among others:

• Independence of ANO (especially in regard to decision-making process, i.e the accreditation/assessment decisions should not be influenced by any third body, including MoE);
• Follow-up procedures and activities during review cycles.

In addition, it is necessary to train institutions on internal quality assurance, including self-evaluation, and review experts on external quality assurance.

Creation of databases and IT systems would help to collect and monitor data about study programmes, staff, students and alumni.  

EVALUATION REPORT FOR INSTITUTIONAL PILOT EVALUATION of Azerbaijan State University of Economics

EVALUATION REPORT FOR INSTITUTIONAL PILOT EVALUATION of Azerbaijan State Pedagogical University