dnn模型 list index out of range_基于svm的财务预警模型

前言

本文将我国A股上市公司作为研究对象，选取了A股 2015-2019 年度被 ST 或被 *ST上市公司，剔除了部分非财务原因导致ST或*ST的上市公司。财务指标选择了T-3期的资产负债率、流动比率、应收账款周转率等10个财务指标。

数据收集及预处理

导入所需要库

import tushare as tspro = ts.pro_api('*')  #tushre的tokenimport pandas as pdimport time

通过tushare.pro取得上市公司信息列表

data_Lcompany=pro.stock_basic(exchange='', list_status='L', fields='ts_code,symbol')data_Dcompany=pro.stock_basic(exchange='', list_status='D', fields='ts_code,symbol')data_Pcompany=pro.stock_basic(exchange='', list_status='P', fields='ts_code,symbol')data_company=pd.concat([data_Lcompany,data_Dcompany,data_Pcompany]

读取CSMAR数据库导出的样本数据并做数据预处理

data_sample=pd.read_excel(r"C:\Users\hasee\Desktop\样本.xlsx")data_sample.columns=data_sample.iloc[0]data_sample.drop(index=0,inplace=True)data_sample.loc[:,"执行日期"]=pd.to_datetime(data_sample.loc[:,"执行日期"])data_sample["年份"]=pd.DatetimeIndex(data_sample.loc[:,"执行日期"]).yeardata_sample=data_sample[["证券代码","年份"]]data_sample.columns=["symbol","年份"]data_sample=pd.merge(data_sample,data_company,how='left',on="symbol"

对样本数据进一步预处理，与正常公司数据合并，通过tushare.pro读取财务指标并写入csv

year=[2015,2016,2017,2018,2019]createVar = locals()for j in year:    createVar["data_sample_"+str(j)]=data_sample[data_sample["年份"]==j]    createVar["data_origin_"+str(j)]=pd.merge(data_Lcompany,createVar["data_sample_"+str(j)],how='outer')                                         createVar["data_origin_"+str(j)]["label"]=createVar["data_origin_"+str(j)]["年份"].map({j:0})                                            createVar["data_origin_"+str(j)].fillna(1,inplace=True)    print(j)                                              del createVar["data_origin_"+str(j)]["年份"]    for i in range(createVar["data_origin_"+str(j)].shape[0]):        print(i)        if i==0:            createVar["fina_indicator_"+str(j)]=pro.query('fina_indicator', ts_code=createVar["data_origin_"+str(j)]["ts_code"][i], start_date=str(j-3)+"1231", end_date=str(j-3)+"1231")        else:            time.sleep(0.9)            createVar["fina_indicator_"+str(j)]=pd.concat([createVar["fina_indicator_"+str(j)],pro.query('fina_indicator', ts_code=createVar["data_origin_"+str(j)]["ts_code"][i], start_date=str(j-3)+"1231", end_date=str(j-3)+"1231")])     createVar["fina_indicator_"+str(j)].to_csv("fina_indicator_"+str(j)+".csv")

读取财务指标选取财务指标并做数据预处理

fina_indicator_2015=pd.read_csv("fina_indicator_2015.csv")del fina_indicator_2015["Unnamed: 0"]fina_indicator_2016=pd.read_csv("fina_indicator_2016.csv")del fina_indicator_2016["Unnamed: 0"]fina_indicator_2017=pd.read_csv("fina_indicator_2017.csv")del fina_indicator_2017["Unnamed: 0"]fina_indicator_2018=pd.read_csv("fina_indicator_2018.csv")del fina_indicator_2018["Unnamed: 0"]fina_indicator_2019=pd.read_csv("fina_indicator_2019.csv")del fina_indicator_2019["Unnamed: 0"]data_origin_2015=pd.merge(fina_indicator_2015,data_origin_2015,on="ts_code",how='left')data_origin_2016=pd.merge(fina_indicator_2016,data_origin_2016,on="ts_code",how='left')data_origin_2017=pd.merge(fina_indicator_2016,data_origin_2017,on="ts_code",how='left')data_origin_2018=pd.merge(fina_indicator_2016,data_origin_2018,on="ts_code",how='left')data_origin_2019=pd.merge(fina_indicator_2016,data_origin_2019,on="ts_code",how='left')data_origin=pd.concat([data_origin_2015,data_origin_2016,data_origin_2017,data_origin_2018,data_origin_2019])data_origin.index=range(len(data_origin))data_origin=data_origin[["debt_to_assets","fcff","profit_dedt","current_ratio","ar_turn","profit_to_gr","roa","grossprofit_margin","or_yoy","ebt_yoy","assets_yoy","label"]]data_origin["fcff_ratio"]=data_origin["fcff"]/data_origin["profit_dedt"]data_origin.dropna(axis=0, how='any',inplace=True)data_origin.index=range(len(data_origin))label=data_origin["label"]data_origin.drop("fcff", axis=1,inplace=True)data_origin.drop("profit_dedt", axis=1,inplace=True)

模型调参

导入所需的库

from sklearn.svm import SVCfrom sklearn.model_selection import train_test_splitfrom sklearn.model_selection import cross_val_scorefrom sklearn.metrics import roc_auc_score, recall_scorefrom sklearn.metrics import roc_curvefrom sklearn.metrics import roc_auc_score as AUCfrom time import time import datetimefrom sklearn.preprocessing import StandardScalerimport numpy as npimport matplotlib.pyplot as plt

划分训练集测试集

Xtrain, Xtest, Ytrain, Ytest = train_test_split(data_origin,label,test_size=0.3,random_state=400)

为什么要划分训练集测试集，请看这里。

https://www.zhihu.com/question/22872584

数据归一化

ss = StandardScaler()ss = ss.fit(Xtrain)Xtrain = ss.transform(Xtrain)Xtest = ss.transform(Xtest)

选择核函数

times = time() for kernel in ["linear","poly","rbf","sigmoid"]:    clf = SVC(kernel = kernel              ,gamma="auto"              ,degree = 1              ,cache_size = 3000              ,class_weight = "balanced").fit(Xtrain, Ytrain)    result = clf.predict(Xtest)    score = clf.score(Xtest,Ytest)    recall = recall_score(Ytest, result)    auc = roc_auc_score(Ytest,clf.decision_function(Xtest))    print("%s 's testing accuracy %f, recall is %f', auc is %f" % (kernel,score,recall,auc))    print(datetime.datetime.fromtimestamp(time()-times).strftime("%M:%S:%f"))

因为训练集中正常公司远远多于非正常公司，所以存在样本不均衡问题。分类模型天生会倾向于多数的类，让多数类更容易被判断正确，少数类被牺牲掉。但是这样对我们的模式是没有意义的，根本无法达到我们的“要识别几年后会处于财务困境的公司”的建模目的。 SVC中class_weight中"balanced"

参数可以比较好地修正我们的样本不均衡情况。

返回结果

返回了三个指标Accuracy、Recall、AUC面积。

准确率Accuracy就是所有预测正确的所有样本除以总样本，通常来说越接近1越好。

召回率Recall又被称为敏感度(sensitivity)，真正率，查全率，表示所有真实为1的样本中，被我们预测正确的样本所占的比例。召回率越高，代表我们尽量捕捉出了越多的少数类，召回率越低，代表我们没有捕捉出足够的少数类。

AUC面积反应了模型捕捉少数类的多数类之间的平衡能力。AUC面积与ROC曲线有关系后面会讲。

在这里我们选择AUC面积作为选择核函数的指标，rbf核函数auc面积最大，选择rbf核函数(高斯径向基核函数)。

对Gamma调参

recallall = []aucall = []scoreall = []gamma_range = np.logspace(-10,1,20)for i in gamma_range:    clf = clf = SVC(kernel = "rbf"              ,gamma=i              ,degree = 1              ,cache_size = 3000              ,class_weight = "balanced").fit(Xtrain, Ytrain)    result = clf.predict(Xtest)    score = clf.score(Xtest,Ytest)    recall = recall_score(Ytest, result)    auc = roc_auc_score(Ytest,clf.decision_function(Xtest))    recallall.append(recall)    aucall.append(auc)    scoreall.append(score)print(max(scoreall), gamma_range[scoreall.index(max(scoreall))])print(max(aucall), gamma_range[aucall.index(max(aucall))])print(max(recallall), gamma_range[recallall.index(max(recallall))])plt.plot(gamma_range,scoreall)plt.plot(gamma_range,aucall)plt.plot(gamma_range,recallall)plt.show()

返回结果

可以看出在Gamma取0.04832930238571752时AUC面积达到了0.757。

对软间隔指标C调参

recallall = []aucall = []scoreall = []C_range = np.linspace(0.01,30,50)for i in C_range:    clf = SVC(kernel = "rbf",C=i,gamma=0.04832930238571752                  ,cache_size = 3000                  ,class_weight = "balanced").fit(Xtrain, Ytrain)    result = clf.predict(Xtest)    score = clf.score(Xtest,Ytest)    recall = recall_score(Ytest, result)        auc = roc_auc_score(Ytest,clf.decision_function(Xtest))    recallall.append(recall)    aucall.append(auc)    scoreall.append(score)print(max(scoreall), C_range[scoreall.index(max(scoreall))])print(max(aucall), C_range[aucall.index(max(aucall))])print(max(recallall), C_range[recallall.index(max(recallall))])plt.plot(C_range,scoreall)plt.plot(C_range,aucall)plt.plot(C_range,recallall)plt.show()

返回结果

可以看出在C取0.6220408163265306时AUC面积达到了0.758。

因为Poly核函数可调的参数更多，我还尝试用网格搜索对poly核函数调参，奈何跑了一天出来的结果也不理想。

绘制ROC曲线

clf_proba = clf = SVC(kernel = "rbf",C=0.6220408163265306,gamma=0.04832930238571752                  ,cache_size = 3000                  ,class_weight = "balanced").fit(Xtrain, Ytrain)FPR, recall, thresholds = roc_curve(Ytest,clf_proba.decision_function(Xtest), pos_label=1)area = AUC(Ytest,clf_proba.decision_function(Xtest))plt.figure()plt.plot(FPR, recall, color='red',         label='ROC curve (area = %0.2f)' % area)plt.plot([0, 1], [0, 1], color='black',linestyle='--')plt.xlim([-0.05, 1.05])plt.ylim([-0.05, 1.05])plt.xlabel('False Positive Rate')plt.ylabel('Recall')plt.title('Receiver operating characteristic example')plt.legend(loc="lower right")plt.show()

ROC曲线，全称The Receiver Operating Characteristic Curve，译为受试者操作特性曲线。这是一条以不同阈值下的假正率FPR(假正率就是一个模型将多数类判断错误的能力)为横坐标，不同阈值下的召回率Recall为纵坐标的曲线，在我们这个案例中阈值为每个样本到划分数据集的超平面的距离。而AUC面积就是ROC曲线下的面积。

关于ROC曲线更多的知识请看这里：

https://blog.csdn.net/dujiahei/article/details/87932096

返回结果

绘制最佳阈值在ROC曲线中的位置

max((recall - FPR).tolist())maxindex = (recall - FPR).tolist().index(max(recall - FPR))plt.figure()plt.plot(FPR, recall, color='red',         label='ROC curve (area = %0.2f)' % area)plt.plot([0, 1], [0, 1], color='black', linestyle='--')plt.scatter(FPR[maxindex],recall[maxindex],c="black",s=30)plt.xlim([-0.05, 1.05])plt.ylim([-0.05, 1.05])plt.xlabel('False Positive Rate')plt.ylabel('Recall')plt.title('Receiver operating characteristic example')plt.legend(loc="lower right")plt.show()

返回结果

基于我们选出的最佳阈值，我们来人为确定y_predict，并确定在这个阈值下的recall和准确度的值。

prob = pd.DataFrame(clf_proba.decision_function(Xtest))prob.loc[prob.iloc[:,0] >= thresholds[maxindex],"y_pred"]=1prob.loc[prob.iloc[:,0] < thresholds[maxindex],"y_pred"]=0prob.loc[:,"y_pred"].isnull().sum()times = time()score = AC(Ytest,prob.loc[:,"y_pred"].values)recall = recall_score(Ytest, prob.loc[:,"y_pred"])print("testing accuracy %f,recall is %f" % (score,recall))print(datetime.datetime.fromtimestamp(time()-times).strftime("%M:%S:%f"))

返回结果