ANICT

Towards a sustainable research career with progression based on merit

IF2013 – Evaluation results

The results for the IF2013 call are now available on the IF FCT website at https://if.fct.pt/.

Joao Lopes
ANICT

Advertisements

33 responses to “IF2013 – Evaluation results

  1. Franklin December 10, 2013 at 8:21 pm

    FCT sent me the comments and scores of only 2 external reviewers both of them awarded points that would allow me to get the position. However, the overall score awarded by the panel is only 6, which is 2 points less than the reviewers awarded. Do you have any idea? Note: The comments and score of Portuguese reviewer who assisted the panel ranking is missing.

    The comments of the evaluation panel reads as follows.
    This evaluation report contains the final score awarded by the review panel. The discussion of the panel was conducted within the context of the reports submitted by the external reviewers. The panel closely analyzed these reviews and, “while not necessarily subscribing to each and every opinion expressed, found that they provide a fair overall assessment”. Based on these reviews, the panel judged that this application did not reach high enough priority to be funded in the present call.

    Hope you can understand how the internal reviewer must have assisted the panel ranking. God bless you.

  2. Andrea Zille December 10, 2013 at 9:05 pm

    Same for me. Two external reviewer. An average of 7.2 and a final overall score of 6. Neither information about the Portuguese reviewer nor about how the final score was attributed. Lack of transparency is a rule ….

  3. Cátia Teixeira December 11, 2013 at 12:24 am

    Same for me. Three external reviewers; an average of 7.4 and a final overall score of 6. If the members of the evaluation panel state that the reports of the external reviewers provide a fair evaluation of the proposal, their overall score clearly does not… In such situations, it seems fair to me that they should give an explanation for such discrepancy.

  4. Andrea Zille December 11, 2013 at 11:46 am

    I was wondering if there is a way to know how many applications have been submitted and how many were rejected before evaluation. I have counted 1479 applications in the published list but the higher ID application number is 1822. It seems that around 2000 people have participated.

  5. Alexandra Lopes December 11, 2013 at 11:55 am

    I agree with all of the above. I was evaluated in the “Life Sciences Evaluation Panel” and had 3 “weighted scores”, from 3 independent reviewers: 7.242, 8.458 and 6.033 and an overall classification of 6.

    It is not at all clear to me how:
    1. each weighted score of each reviewer is calculated or how it goes down to decimals. The IFCT 2013 regulation states: “Each of the three components of the application are rated using the 9-point scale with whole numbers only (no decimal ratings).”

    2. is the overall rating calculated. The IFCT regulation states: “The overall rating reflects the Panel´s judgment on the application and does not result from an arithmetic formula applied to the scores given to each component.”
    It is clear that it is not an average but I cannot think of any reasonable objective criteria to end up with an overall rating of 6.

    Moreover, the reviewers only give general sentences to justify each “weighted score” given and therefore I must assume that the whole process is highly subjective.

    For example the main comments that resulted in a classification of 6 were: “The candidate has a good profile, the top 10 publications are excellent and relevant for the project. Overall, the project is very interesting and promising. The work plan is precise and the objectives seem realistic. The background section is an excellent introduction to this project. The writing style is however a little bit surprising.”

    Bear in mind that only applications with an overall rating of 7 or more were evaluated by the actual review panel that decided if the applications would be recommended for funding. Therefore this obscure appreciation process disqualified many applications from review by the international panel.

    I believe it would help if we all shared our results and doubts in order to be more prepared for the audiência prévia. To keep the evaluations in secret will only legitimate the obscurity of this clearly unfair process.

  6. Franklin December 12, 2013 at 6:48 am

    Dear Joao and Colleagues

    FCT can not clarify the questions regarding the final ranking as they have played a discrimination game with the scores awarded by international experts through its proxy panels. The duty of a panel should be the ranking of the candidates by the scores awarded by the experts. The panel dont have any right to play with the scores. For instance, in my case 2 points have been reduced from the average awarded by the experts. If the panel did the same reduction to all applicants, no one would have got selected. Hence, equal treatment is also in question in the process. I suspect a vital role of FCT in the cherry picking game.

    Thanks for your advise regarding the rules of “audiencia previa” to see our process. With my long term experience and by the experience gained from the IF 2012, the issues can not be fixed through this process. If FCT can manipulate the scores of an international call, they can easily trick you in the one-to-one appeal. Finally, you will receive a one sentence reply saying that you appeal is not successful. The only way to deal with this issue is through Portuguese court or European court of justice. As a qualified lawyer, it is my duty to take this issue to a court and get justice.

    Moreover, the issue is not only how the applications of unsuccessful candidates were dealt by the panel, but also how it was dealt with the successful applicants. In this scenario, it is essential that FCT should disseminate to all of us the scores awarded by the three external reviewers to the successful candidates, in addition to the overall score awarded by the panel. This would bring into light, the influence of panel or people from the FCT council or research centres.

    If ANICT is fair and interested in the justice to all the ANICT members, I request your kind co-operation in this matter.

    • iacopo December 12, 2013 at 2:16 pm

      In my case, having got an overall score of 7 I can not even know how and to what extent they might have played with the scores (I guess much higher than 7-8 on average, otherwise I would have got something similar to people with a final 6…) given by the external reviewers, as all my provided evaluation was a six lines message missing any mark apart the “final” one!

  7. Franklin December 12, 2013 at 9:24 am

    FCT makes statement on 2013 FCT Investigator call

    In response to several questions and issues that have been passed on to the FCT Board and departments, relating to the way in which the evaluation process of the 2013 FCT Investigator call was carried out, FCT hereby provides the following clarification:

    A – Reviewers and Evaluation Panel members

    4. To support the panel’s decision, each application was, as a rule, assessed by three different external reviewers, including two non-Portuguese and one Portuguese reviewer. All reviewers have internationally acknowledged scientific merit;
    Q- SO, WHERE IS THE THIRD REVIEWER COMMENT IN MINE AND SOME OTHERS CASE

    5. The use made of the external evaluations during the evaluation process is entirely the responsibility of the panel members. The form and content of such use shall be conveyed to all the applicants as specified by the evaluation panel and articulated in the Evaluation Guide, available from the FCT website;
    Q- IF THE PROCESS IS ENTIRELY ON THE PANEL, WHY THERE IS A NEED FOR PEER-REVIEW. DOES THE GUIDE SUGGEST THE PANEL TO CUT 2 POINTS FROM WHAT THE REVIEWERS AWARDED?

    6. These procedures are similar to those adopted in other recently launched FCT funding calls, namely the call for Exploratory Research Projects in all scientific domains – 2013.

    Q- SO, THE IRREGULARITIES EXISTED EVEN IN THE R AND D FUNDING?

    B – Appeal process (Audiência prévia) and hiring

    1. On 26 November, after receiving the comments of the Evaluation Panel, FCT notified all approved applicants of FCT’s decision to fund the contracts that these researchers may set up with host institutions;
    2. In the same message, FCT notified the applicants that “Under the circumstances of a positive funding decision, the appeal process (audiência prévia) is wavered, under the terms of article 103 (2-b) of the Código de Procedimento Administrativo, and that the selection process for these applicants is thus considered to have been completed;
    3. The right of appeal (as prescribed by law) of all candidates that were not recommended for funding after evaluation by the panel is safeguarded. Furthermore, applicants were informed that the appeal period would only begin when their scores and reviewer and/or panel comments are made available on the online platform;
    4. FCT has made legal and budgetary provisions to allow funding of any non-recommended application that may be recommended for funding after the appeal process;
    5. FCT thus warrants that the call and ancillary procedures concerning those researchers whose applications were not recommended for funding after evaluation will be undertaken in strict compliance with the legal requirements.

    Q- WE WILL RECEIVE THE FOLLOWING RESPONSE AFTER 8 MONTHS OF APPEAL
    Caro Candidato,
    Pedimos desculpa pelo atraso significativo na avaliação dos recursos submetidos no âmbito do concurso Investigador FCT 2012.
    Após o cumprimento do período de submissão de reclamação, encontra-se concluído o procedimento concursal de recrutamento e contratação de doutorados – Programa IF 2012.
    Nesta conformidade informa-se V. Exa que a candidatura apresentada, e os fundamentos aduzidos, em sede de reclamação, não justificam qualquer alteração à avaliação efetuada, tornando-se esta definitiva, nos termos e fundamentos anteriormente notificados e que se dão aqui por reproduzidos.
    O concurso IF 2012 foi extremamente competitivo e esperamos que possa submeter uma nova candidatura em concursos futuros.
    Com os melhores cumprimentos,
    O Conselho Diretivo

  8. Alexandra Lopes December 12, 2013 at 10:18 am

    Another useful information: 959 out of 1479 candidates in the scoring list (67%) had an overall score of 6 (and just missed the chance to reach the evaluation panel).

  9. Alexandra Lopes December 12, 2013 at 11:00 am

    correction: 64%

  10. Teresa Salomé Mota December 13, 2013 at 11:09 am

    My case is very similar to the ones above: three external reviewers with general comments on how good is my CV, my publications and my project; medium final score given 7. The Evaluation Panel gave me 6 don’t know why. One of the external reviewers didn’t even red my call as his (her) comments prove; in fact, there were even lies in his (her) comment.
    And let us be honest and tell something that everyone has in mind but doesn’t say: Cv of most people that were selected is public and we make comparisons. Some of them simply don’t deserve. And this is the reason why the FCT call was the way it was.

      • João Lopes December 13, 2013 at 7:35 pm

        Many comments in this blog still reveal substantial lack of information. Please note for instance that a final score of 6 does not mean that the application was NOT evaluated by the pannel. I won’t enter into further details now because I do not want to provide incomplete/speculative information. ANICT has a meeting with the FCT presidency next tuesday. We are committed to ensure that all questions will receive an appropriate answer. We do not speculate, though. We will get back to you by tuesday with a complete set of informations.
        Joao Lopes/ANICT

      • iacopo December 14, 2013 at 12:49 pm

        Not exact, Joao: everybody has already understood that the application was (somehow) actually evaluated (although this term, as normally intended, likely does not accurately express and represent what it has been really done) by the panel, unfortunately… Overall, any lack of information and any misundertsanding could be easily overcome by providing true, complete and clear information, but this would be clearly not convenient for a number of people.

  11. Franklin December 13, 2013 at 9:16 pm

    Hello Joao

    I believe that obtaining CLEAR answers for the following questions from FCT will be useful for us

    1. What is the minimum number of reviewers for each application? whether the number of reviewers for all candidates are equal?

    2. Why FCT sent the comments of 2 reviewers for some candidates and 3 or more for others?

    3. How the final score was reached from the scores awarded by the reviewers? e.g. some candidates got more than the reviewers awarded and some people got less why? was there any undisclosed criteria?

    4. Panel assessment of my application says “the panel judged that this application did not reach high enough priority to be funded in the present call”
    What did they mean by PRIORITY?

    5. How some specific groups/ labs (I dont mean centres) got 4-5 positions?

    6. How the initial number of selected candidates 204 became 210 at present?

    7. How many positions are expected to be funded in the appeal process?

    8. How the appeals will be handled, who will be responsible?

    All the best for the meeting and have a nice weekend

    • iacopo December 14, 2013 at 9:38 am

      Just to tragically smile:

      1. The minimum is two, particularly because in most cases three external reviewers were contacted, but one did not even reply. And he/she did well as their evaluation was intended to be finally ignored/reversed. Thus, the number of evaluators was different for different candidates, but no problem as in the next World Cup some matches will be ruled by three referees, other by two.
      2. Already answered at point 1
      3. yes, there were tons of undisclosed criteria, but you can not know, otherwise everything would be fair and boring
      4. They mean that it was clearly anticipated in a paper they have in their bathrooms that funding was not designated to individual researchers and proposals themselves, but to priority areas. Of course, knowing in advance such priority areas, it is your fault to have taken part to the call anyway. However, our congratulations as the importance is not to win, but to participate, thus you are great, according to De Coubertin.
      5. It is not because those groups are politically stronger, but because the do priority research. Of course, the priority research would be safe even funding just one or two people working on the same topic, but it was clearly explained since the beginning that the key aspect is the individual researcher, thus it would have been unfair to penalize a number of deserving people for this reason.
      6. Because maybe some people were then excluded as they could not legally take part to the call due to some specificity of their current contracts/positions (or due to any other reason), then some people from the promptly communicated reserve list “climbed up”. Or maybe some proper phone call arrived meanwhile.
      7. Most likely zero as (1) everything was done perfectly and (2) point 8 applies.
      8. The same people who arbitrarily decided on funding vs rejection will decide on the appeals, that will be carefully handled through a two-stage process: the first during the Christmas lunch, the second during the penalties of the World Cup Final in Brazil. But no worries, they will take each argument into account and will not miss anything out of the clearly illustrated criteria for that.

      • Franklin December 14, 2013 at 10:02 am

        Having already spent 10 years here, now I really regret for my stupid decision to come to Portugal from USA.

        I tried google translate, but didn’t get the correct meaning. Could anyone help me translating the following phrase?

        «Tudo isto é um sistema integrado, não existe ciência isolada num investigador», afirmou Miguel Seabra e acrescentou que «queremos é criar as condições para que esses investigadores estejam em centros de excelência, que esses centros de excelência se rodeiam de infraestruturas de excelência e que essas infraestruturas ou que o sistema nacional também esteja em direto contato internacional e fazer parte de uma rede internacional de centros de excelência nas respetivas áreas».

        Thanks

  12. Joao_A_Lopes December 14, 2013 at 1:20 am

    Thanks a lot for the query list. We’ll try to get answers for all these issues.
    Have a nice weekend also!

  13. Andrea Zille December 14, 2013 at 1:47 am

    I have knowledge of at least two colleagues with a final score of 7 who had no reviewer’s comments at all but only the evaluation panel comments without any information about the individual reviewer scores.

  14. Isaac Vidana December 14, 2013 at 2:25 pm

    Also in my case I received the evaluation of the 3 external reviewers which in average was
    7.4 but on the basis, as it is said in the report of the evaluation panel, became a final score
    of 6.

    I put here the paragraph that apparently is the report of the evaluation panel. I now that many
    other people got exactly the same, and when I say exactly the sameI really mean EXACTLY THE
    SAME. I am in the area of Physical Sciences and I now that people of completely different
    areas like Social Sciences also got THE SAME TEXT.

    That is simply unacceptable. A common text provided to many people cannot never be considered
    the report of ANY application.

    This is the text I received, please check whether it is the same you got or very similar. This is
    a clear proof that the evaluation panel never based his decision on (as said in the Evaluation
    Guide) the FULL APPLICATION of the candidate and the reports of the 3 experts.

    »»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»
    Physical Sciences and Engineering Evaluation Panel – IF 2013
    »»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»
    This evaluation report contains the final score awarded by the review panel. The discussion of the panel was conducted within the context of the reports submitted by the external reviewers. The panel closely analyzed these reviews and, while not necessarily subscribing to each and every opinion expressed, found that they provide a fair overall assessment. Overall the panel considers this proposal to be of reasonably good quality. However based on the combined set of criteria used in the assessment it was not ranked high enough to be funded.

    • Franklin December 15, 2013 at 9:05 am

      Isaac! I received the following, but, I found that several people got the same like mine!!

      This evaluation report contains the final score awarded by the review panel. The discussion of the panel was conducted within the context of the reports submitted by the external reviewers. The panel closely analyzed these reviews and, while not necessarily subscribing to each and every opinion expressed, found that they provide a fair overall assessment. Based on these reviews, the panel judged that this application did not reach high enough priority to be funded in the present call.

      • Isaac Vidana December 16, 2013 at 8:26 am

        You see the first 4 sentences are the same also in your case. I checked this with several people of my area, in all the cases these 4 sentences are always the same and in many cases the complete report is the same. So another question to ask FCT is how is possible that a single report is provided to many people that are working in different fields ?. It seems that they gave “coffee for everybody”, but this is not a solution.

        • Franklin December 16, 2013 at 9:11 am

          Yes. You made the point. I hope Joao will ask this question also.

          If you go to this page http://empregocientifico.blogspot.pt/p/avaliacoes-da-fct.html
          you will find even more.

          I found 17/25 matches across different evaluation panels by simply doing ctrl+f

          “the so-called informatics problem didn’t provide enough time to type individual comments to all” if you want individual comments, they would need another informatics problem, but it will not be difficult.

          Ha ha ha…

          What a great international evaluation!!!!

  15. Franklin December 15, 2013 at 7:26 pm

    FEW MORE QUESTIONS:

    1. What was the cut-off score (reviewer weighted) of applications that reached the International panel?

    2. Did all applications which obtained more than 7 average weighted reviewer score reach the International panel?

    3. On what criteria and who awarded the final overall score? I mean, whether the scoring was done by the International panel or it was done before reaching the panel? if it was done before reaching the panel, who was the “secret mediator”?

    4. If there was a “secret mediator”, you need to ask the FCT presidency to make the list public

    Thanks

    • Isaac Vidana December 16, 2013 at 10:15 am

      I’ll tell you how I think I went. The 3 external (an still UNKNOWN) reviewers saw all the applications. They made a CUT. Those BELOW an average value of 7 WERE REJECTECTED and were NEVER saw by the Evaluation Panel. THOSE WITH MORE THAN
      7 reach the panel. HOWEVER, contrary with what is written in the EVALUATION GUIDE, the
      panel NEVER REVIEW THE FULL APPLICATIONS, it bases its decision on the “REPORTS”
      (let me put in question marks) of the 3 external guys. According to some, as they mention,
      “COMBINED SET OF CRITERIA” (I still wonder which criteria) they RANDOMLY gave
      6 or 7 votes to the EXCLUDED peopla and send them send the SAME REPORT. That’s
      probably the full story. BUT of course, FCT still insist (and will insist) on the INTERNATIONAL CARACTER of this COMEDY as if that was a guarantee of the cleaness
      of the process.

      • Franklin December 16, 2013 at 10:48 am

        I agree with you that the unknown reviewers (probably 2) saw the applications. But, they did not make the CUT, they just sent the average weighted score with comments. Then, it came to the hands of the GOD’s, “the so-called national evaluators”, who made the third evaluation (probably) and made the overall score, as instructed without worrying about the external reviewer scores, which determined the overall process. This is the place where, the overall score was reduced or increased in a biased manner. Then, candidates with overall score above 7 reached the panel. Panel did not make any change in the list. Automatically, candidates with 8 and 9 got the positions, as determined by the GOD. Here, nothing to blame the Foreign reviewers or the international panel. They did their Job. However, the million dollar question is the identity of the GODS, which still remain secret. The overall evaluation process is little different from what happened in IF2012 though. In IF2012, the GOD filtered the candidates in a biased manner before reaching the reviewers but in IF2013, the GOD acted after the reviewers. The pattern of evolution looks like in IF2014, the GOD might act after the panel ranking. ALTHOUGH THE EVALUATION PROCESSES WILL BE LITTLE DIFFERENT, THE GOD WILL MAKE SURE THE OUTCOME IS SAME IN ALL CALLS. ONLY THING WE CAN DO IS PRAY.

  16. Franklin December 16, 2013 at 11:17 am

    I THINK IF.FCT.PT AGAIN ENCOUNTERED INFORMATICS PROBLEM. I AM UNABLE TO LOGIN. PLEASE LET ME KNOW, IF ANYONE HAVE THE SAME PROBLEM

  17. Isaac Vidana December 16, 2013 at 8:16 pm

    Unfortunately, we’ll never know how things really went in this obscure process. FCT will continue claiming that everything went according the rules. The point is that we do not know and never will now the rules of this game.

  18. Alexandra Lopes December 17, 2013 at 11:33 am

    Dear João Lopes,

    Thank you for your commitment to make this a more transparent process. Regarding your observation on our lack of information and level of speculation let me bring this to your attention:

    According to one of the reviewers in the Life Sciences panel only 40% of the applications were actually evaluated by the panel (please see Carl-Henrik Heldin interview in Publico in 6/12/2013). Even though we do not have access to the number of applicants with each score in each area, the overall number of candidates with a final score of 6 was approximately 60% of the total. It therefore seems very unlikely that all applications with final score of 6 in Life Sciences have been EVALUATED by the international panel that Carl-Henrik Heldin was part of. This is not speculation, it is simple maths.
    Intriguingly one of the candidates who published his evaluation report got this information by the review panel: “This application was not discussed in detail by the panel since it was scored below the panel threshold for discussion (average of external reviews below 6.5). The panel advises that, in accordance to the evaluation guidelines, this application should receive all scores and comments from mail reviewers. The overall score is the average of the external reviewers’ weighted scores.”
    But many applicants with and average weighted score >6.5 did not get any comments from the evaluation panel and therefore these applications were also not discussed in detail by the panel.

    Therefore we all must request in audiência prévia the acta of the meeting of the evaluation panel were the applications were discussed/evaluated. In that way we will be able to see what actually was said and done about each application, irrespective of what FCT may publicly state about it.

    • iacopo December 17, 2013 at 2:59 pm

      Try yourself to arbitrarily “evaluate” more than 1400 “units” (whatever) and then provide consistent documents… in order to be convincing, you will need to be an evil genius, but no doubt that here the first is true, while the second not even close!!
      Thus, their impossibility to provide any formal, clear and complete report is obvious.

      • Franklin January 13, 2014 at 10:33 pm

        Have anyone with overall score of 7 received your reviewer comments? FCT already made the documents regarding the evaluation available.

  19. Leitor incrédulo, Algures neste país desgovernado December 19, 2013 at 1:39 pm

    I am *very* surprised since newer posts about this issue can not be commented. So ANICT claims the IF application has no problems at all and doesn’t accept to discuss this.
    This is completely unbelievable.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: