Forskningsanalyse? Kvalitetsindikatorer? Hvad er det for noget?

Præsentationer af emnet: "Forskningsanalyse? Kvalitetsindikatorer? Hvad er det for noget?"— Præsentationens transcript:

1 Forskningsanalyse? Kvalitetsindikatorer? Hvad er det for noget?
Jeppe Nicolaisen Lektor, Ph.d. Danmarks Biblioteksskole

2

3

4

5 Det kan forekomme nærved blasfemisk at påstå, at man kan sige noget centralt om videnskab bare ved at tælle antal sider eller antal tidsskrifter på et felt. Er man ikke nødt til at læse afhandlingerne, for at se om de er noget værd? Tor Nørretranders (2007, s. 126)

6 Fremover bør vi anlægge internationale, objektive kriterier for tildeling af forskningsmidler. Det vil bringe dansk forskning op i verdensklasse. Objektiv, international vurdering af forskningens kvalitet gør det meget nemt at kende excellent forskning fra den jævne forskning. Børge Grønne Nordestgaard (2007, s. 9)

7 Bibliometri All studies which seek to quantify processes of written communication Pritchard (1969, s. 348)

8 Bibliometriens anvendelser
Informationssøgning - Referencer, citationer, links Vidensorganisation - Klyngeanalyse, bibliometriske kort Forskningsevaluering - Forfattere, tidsskrifter, institutioner, lande - ’Collection management’

9 Kvalitetsindikatorer
Publikationsanalyse Citationsanalyse

10 Publikationsanalyse Kvantitative målinger på dokumentsamlinger.
Oftest optællinger af indholdet i et eller flere af de bibliografiske felter i en afgrænset del af en bibliografisk database. Bl.a. optællinger af forfatteres, institutioners eller landes videnskabelige produktion fordelt på emne, tidsskrifter osv.

11 Citationsanalyse Kvantitative studier af dokumenters litteraturlister
Man skelner mellem to former for citationsanalyser: - Referencer fra en samling dokumenter - Citationer til en samling dokumenter

12

13 Bradfords lov En type af matematisk-statistisk fordeling, opstillet på grundlag af iagttagelser over spredningen af artikler om et givent emne i tidsskrifter - nemlig at nogle få af tidsskrifterne rummer hovedparten af artiklerne, f.eks: 8 tss. producerer 110 artikler 29 tss. producerer 133 artikler 1: 4: 16 127 tss producerer 152 artikler }

14

15 Bradfords lov Management of Serials in Libraries:
… de følgende pointer er nogle af de mest åbenbare potentialer af Bradford analyser: - Definition af emnets kerne - Selektion/kassation - Samlingsevaluering - Beregning af omkostninger ved forskellig dækningsgrad - Prioritering af tidsskrifter Nisonger (1998, s )

16 Bradfords lov Problemer (bl.a.): Kvantitet/kvalitet
Definitionen af ”emne” - Ikke tidligere diskuteret ifht. Bradford’s lov Nicolaisen, J. & Hjørland, B. (2007) Practical potentials of Bradford's law: A critical examination of the received view. Journal of Documentation, 63(3): Hjørland, B. & Nicolaisen, J. (2005). Bradford's law of scattering: ambiguities in the concept of "subject". Proceedings of the 5th International Conference on Conceptions of Library and Information Science:

17 Hvornår er noget videnskabeligt?
Price (1970): 0 - 9 referencer…….. Unscholarly referencer.….. Scholarly 23+ referencer……..... Non-creative scholarship

18 Hvornår er noget videnskabeligt?
Paul Ehrlich (1909): Salvarsan Alexander Fleming (1929): Penicilin

19 Hvornår er noget videnskabeligt?
Disciplinafhængig (Hyland, 2000): Discipline Average per paper words Sociology 104,0 12,5 Marketing 94,9 10,1 Philosophy 85,2 10,8 Molecular biology 82,7 15,5 Applied linguistics 75,3 10,8 Electronic engineering 42,8 8,4 Mechanical engineering 27,5 7,3 Magnetic physics 24,8 7,4

20 Den første citationsanalyse
Consideration of the method of investigation [citation counts] here employed will show that we are concerned not merely with the quantity of work published […], but that in reality we are concerned only with the good work, the work which has survived and which has proved of value to the investigators who followed. (Gross 1927, s. 641) Citationer = Kvalitet

21 Citationsdata generelle problemer
Seglen (1996) [translated from Norwegian]: References are selected because of their usefulness for the author, which is something different from their quality  Only a small fraction of all used material is cited General knowledge is not cited Knowledge is often cited from secondary sources Documents supporting an authors arguments are cited more often than other documents Flattering (citing editors, potential referees and other authorities) Show-of (citing hot new “in”-articles) Reference copying (references provided by other authors) Conventions. In biochemistry, for example, methods are cited but not reagents Self citations

22 Bibliometriens forsvar
Small (1987, s. 339): The issue is not whether we can rely on reference lists in individual cases as complete sets of influences (we cannot), but rather whether references can be used statistically, in the aggregate, as an indicator of influence. Nederhof & Van Raan (1987, s. 326): … by no means the assumption is necessary, that scientists cite in their papers all work used in their research but still, citations can be used to monitor scientific influence.

23 Bibliometriens forsvar
Cawkell (1976, s. 3): Citation anomalies [in sence of ”excessive self-citations, plagiarism of references, careless or omitted references etc.] have little effect - they are like random noise in the presence of strong repetitive signals.

24 Gennemsnitsmantraet Citation analysts assume that the biases and deficiencies of individual citers are repaired to a tolerable degree by the combined activity of the many. White (2001, s. 102)

25 Gennemsnitsmantraet Altså:
Hvis citationsanalyser foretages på store datamængder vil man kunne eftervise en nøje sammenhæng mellem kvalitet OG antal citationer. Jo flere citationer – jo større kvalitet… Citationer Kvalitet

26 Bornsteins hypotese If a relationship between citation frequency and research quality does exist, this relationship is not likely to be linear. The relationship between research quality and citation frequency probably takes the form of a J-shaped curve. (Bornstein 1991, s. 31)

27 Citations- frekvens Ekspertvurdering

28 J-kurve? En samling dokumenter En samling ekspertvurderinger af 1
En samling citationer til 1 Stikprøven: Bøger anmeldet i tidsskriftet Contemporary Sociology mellem Ratings fra databasen Sociological Abstracts Citationer søgt i Social Science Citation Index (5-års citationsvindue)

29 J-kurve? Stikprøve: Rating Antal anmeldelser i CS 5% Stikprøve ( ) Very favorable , Favorable , Neutral , Unfavorable , Very unfavorable , Total ,

30 J-kurve? Gennemsnit og median citationsfrekvenser
Rating Gn.snit Median Very Favorable 17,53 17,50 Favorable 19, ,00 Neutral 15, ,00 Unfavorable 8, ,50 Very unfavorable 15, ,00 Total 17, ,00

31 Resultater (gennemsnit)

32 Resultater (median) Nicolaisen, J. (2002).
The J-shaped distribution of citedness. Journal of Documentation, 58(4): Resultater (median)

33 Indikatorteori The current holy grail in scientometrics is the development of indicator theories rather than citation theories. Van der Veer Martens (2001, online)

34 Paul Wouters (1999) The quest for a citation theory it is a dead end (s. 211). Science and technology indicators create a “formalized representation” of science which initially neglects meaning. Of course, to interpret these representations one needs to attribute meaning again. The main point is, however, that this attribution of meaning can be postponed (s. 209).

35 Fred Dretske (1981) Informationel semantik
Et signal bærer betydningen p hvis og kun hvis signalet indikerer p. Røg Brand!

36 The error problem What it indicates is mostly not what it represents…
Tænk på en lille fugl!

37 Sikker mekanisme/ Sikkerhedsmekanisme
Informational semantics seeks to uncover signals, which are reliably correlated with specific situations and hence indicate these situations. In other words, informational semantics attempt to establish certain mechanisms, but only safe ones are logically possible. Nicolaisen (2007, s. 631)

38 Eugene Garfield (1979) A highly cited work is one that has been found useful by a relatively large number of people, or in a relatively large number of experiments […]. The citation count of a particular piece of scientific work does not necessarily say anything about its elegance or its relative importance to the advancement of science or society […]. The only responsible claim made for citation counts as an aid in evaluating individuals is that they provide a measure of the utility or impact of scientific work. They say nothing about the nature of the work, nothing about the reason for its utility or impact (s. 246).

39 Fortolkning Citation analysis is not a shortcut to be used as a substitute for thinking (f.eks. Egghe & Rousseau, 1990, s. 226; Garfield, 1985, s. 408)

40 Introduction to Informetrics (1990)
We expect this book to be of help to the informetrics teacher in organising his or her course and to be interesting and useful both as a course book and as a background reading for students in library and information science (Egghe & Rousseau 1990, s. v).

41 Fortolkning Citationsteori

42 Kulturel disciplinering
Begreber, mening, viden, informationsbehov, relevanskriterier, etc. formes i diskursfællesskaber, f.eks. i videnskabelige discipliner, fag, domæner.

43 Domæneanalyse Forstå domænets - Ontologi - Epistemologi - Sociologi

44 Fortolkningens grundlag
Videnskabsfilosofi Videnskabshistorie Videnskabssociologi

45 Referencer Bornstein, R.F. (1991). The predictive validity of peer review: A neglected issue. Behavioral and Brain Sciences, 14(1): Cawkell, A.E. (1976). Understanding science by analysing its literature. The Information Scientist, 10(1): 3-10. Dretske, F. (1981). Knowledge and the Flow of Information. Cambridge, MA: MIT Press. Egghe, L. & Rousseau, R. (1990). Introduction to Informetrics: Quantitative Methods in Library, Documentation and Information Science. Amsterdam, NL: Elsevier. Garfield, E. (1979). Citation Indexing: Its theory and Applications in Science, Technology and Humanities. New York, NY: Wiley. Garfield, E. (1985). Uses and misuses of citation frequency. Current Contents, 43 (October 28), 3- 9. Gross, P.L.K. (1927). Fundamental science and war. Science, 66: Hyland, K. (2000). Disciplinary Discourses: Social Interactions in Academic Writing. Harlow, UK: Pearson Education Ltd. Martens, B. Van der Veer. (2001) Do citation systems represent theories of truth? Information Research, 6(2). [Available at: Visited September 18, 2005. Nederhof, A.J. & Van Raan, A.F.J. (1987). Citation theory and the Ortega hypothesis. Scientometrics, 12(5-6):

46 Nordestgaard, B. G. (2007). Et strejf af nepotisme [Kronik]
Nordestgaard, B. G. (2007). Et strejf af nepotisme [Kronik]. Jyllands-Posten, 27. august. Nicolaisen, J. (2007). Citation analysis. Annual Review of Information Science and Technology, 41: Nisonger, T.E. (1998). Management of Serials in Libraries. Englewood, CO: Libraries Unlimited. Nørretranders, T. (2007). Civilisation 2.0: Miljø, fællesskab og verdensbillede i linkenes tidsalder. [Frederiksberg]: Thaning & Appel. Price, D.J.S. (1970). Citation measures of hard science, soft science, technology, and nonscience. In: Nelson, C.E. & Pollock, D.K. (eds.), Communication Among Scientists and Engineers. Lexington, MA: Heath: 3–22. Pritchard, A. (1969). Statistical bibliography or Bibliometrics?. Journal of Documentation, 24: Seglen, P.O. (1996). Bruk av siteringer og tidsskriftimpaktfaktor til forskningsevaluering. Biblioteksarbejde, 48: Small, H. (1987). The significance of bibliographic references. Scientometrics, 12(5-6): White, H.D. (2001). Authors as citers over time. Journal of the American Society for Information Science and Technology, 52(2): Wouters, P. (1999). The Citation Culture. Amsterdam, NL: Universiteit van Amsterdam. Ph.D.

47 Download slideshow